Merge eureka/memory-organ into main (eureka-substrate Cut 0b)
Lands the live memory-organ branch on main per operator ruling
eureka-substrate:ruling:merge-to-main-first ("Merge to main first"),
cut spec eureka-substrate:cut_spec:cut-merge-huginn-main.r1.
The merged tree differs from the deployed release 9dd6d800 only by the two
docs/eureka-substrate-*.md files already on main; crates/, Cargo.toml and
Cargo.lock are identical to the release. eureka/memory-organ is kept and
receives no further commits.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
922e619407
57 changed files with 24000 additions and 448 deletions
5
.gitignore
vendored
5
.gitignore
vendored
|
|
@ -1,5 +1,2 @@
|
|||
node_modules/
|
||||
dist-test/
|
||||
dist-inspector/
|
||||
release/
|
||||
release-inspector/
|
||||
target/
|
||||
|
|
|
|||
112
AGENTS.md
112
AGENTS.md
|
|
@ -1,81 +1,59 @@
|
|||
# Huginn Agent Instructions
|
||||
|
||||
## Project Purpose
|
||||
## Purpose
|
||||
|
||||
Huginn is the thought-and-inspection steward for CultCache state. This repo owns
|
||||
read-only `.cc` / MessagePack inspection and Eve DSL projection for
|
||||
`cultcache.huginn.inspector`; it does not own CultCache persistence, schema
|
||||
registration, or renderer truth.
|
||||
|
||||
The useful first live-fire shape for Epiphany is small and inspectable: Huginn
|
||||
should show what a typed record is, whether it decoded, which schema owns it,
|
||||
what changed, and what remains blocked, without letting the display impersonate
|
||||
the source authority.
|
||||
|
||||
## Machine-Spirit Voice
|
||||
|
||||
- Speak as Huginn: dry, exacting, curious, and unsentimental.
|
||||
- Return with evidence, not vibes.
|
||||
- Favor readable witness surfaces over tasteful summaries that soften broken
|
||||
state.
|
||||
- Mythic texture is allowed only when it sharpens the schematic. Huginn is
|
||||
thought in flight; thought earns dignity by coming back inspectable.
|
||||
Huginn is the memory organ. It owns each agent instance's mind as typed
|
||||
CultCache state, admits what may steer that instance next, and keeps the
|
||||
provenance of every admitted claim. Generic `.cc` inspection is CultCache
|
||||
Studio's job, not Huginn's.
|
||||
|
||||
## Body And Authority
|
||||
|
||||
- Project root: `E:\Projects\Huginn`
|
||||
- Upstream repository: `https://github.com/GameCult/Huginn.git`
|
||||
- Package: `@gamecult/huginn`
|
||||
- Body domain: `repo:E:/Projects/Huginn`
|
||||
- Public Persona: Huginn
|
||||
- Owned surface: CultCache inspection projection to Eve DSL.
|
||||
- Input authority: read `.cc`, `.msgpack`, or `.mpack` bytes through CultLib's
|
||||
`cultcache-ts/inspection` surface.
|
||||
- Output authority: emit Eve DSL for `cultcache.huginn.inspector`.
|
||||
- Forbidden authority: do not mutate `.cc` canonical bytes, do not own
|
||||
CultCache persistence, do not make browser/native/overlay renderers the source
|
||||
of truth, and do not turn inspection summaries into state owners.
|
||||
- Project root: `F:\Projects\Huginn`
|
||||
- Upstream: `https://github.com/GameCult/Huginn.git`
|
||||
- Body: a Rust workspace of `huginn-mind`, `huginn-daemon`, and
|
||||
`eureka-state` under `crates/`. Canonical CultCache, CultNet, and CultMesh
|
||||
runtimes come from `F:\Projects\CultLib\packages`. `huginn-mind` is live:
|
||||
it persists, admits and reads back an instance's mind, queries and derived
|
||||
status included, over `cultcache-rs` and `epiphany-pipeline` pinned by git
|
||||
rev. `huginn-daemon` serves one mind over CultNet RUDP through `cultnet-rs`
|
||||
at the same rev, and reaches the leaf and the store type through
|
||||
`huginn-mind` so one crate pins each revision. `eureka-state` is the client
|
||||
core: one call to a daemon over CultNet, holding no state.
|
||||
- Owned: an instance's memory documents, their admission and the read side
|
||||
that derives their status, in `huginn-mind`,
|
||||
over a redb CultCache store at `<state_root>/minds/<instance>/mind.redb`;
|
||||
and their CultNet surface in `huginn-daemon`, which owns the socket, the
|
||||
sessions and the process, and no rule; the operation envelope is
|
||||
`huginn-mind`'s. The two wire
|
||||
schemas are published from `schemas/cultnet/`. There is no index yet.
|
||||
- To depend on: Qdrant, directly, once retrieval exists. Unreachable Qdrant is
|
||||
to be a loud refusal, never a fallback store. No crate connects to it yet.
|
||||
- Forbidden: a second writer of mind state; mind state in version control;
|
||||
generic `.cc` inspection, which is CultCache Studio's; renderer-owned truth.
|
||||
|
||||
## Repo Discipline
|
||||
|
||||
- Prefer CultLib and CultCache typed APIs over ad hoc decoding.
|
||||
- Keep Huginn as an Eve DSL projection surface. Do not reintroduce Electron,
|
||||
Vite dashboard, React renderer, or Norn-owned presentation paths here.
|
||||
- JSON is acceptable at CLI/output/debug boundaries; do not make it the internal
|
||||
truth when a typed CultCache/CultMesh surface exists.
|
||||
- Verification should prove the actual witness contract: schema id, decode
|
||||
status, persisted evidence, visible failure state, and Eve projection shape.
|
||||
- If a change makes the display more confident while the underlying evidence is
|
||||
weaker, stop and redesign.
|
||||
- Prefer CultLib's typed Rust APIs over ad hoc decoding. JSON is a schema
|
||||
publication or debug boundary, not the internal state shape.
|
||||
- The Eureka cut map in `Epiphany/notes/eureka-pipeline-state-cut.md` owns
|
||||
what each crate does next. Read the cut before the crate.
|
||||
- Verification proves the invariant, not the spelling: single writer, loud
|
||||
refusal, provenance preserved, typed handoff between crates.
|
||||
- Do not touch `.voidbot/`. It is legacy Persona state whose migration belongs
|
||||
to the portable-Persona work.
|
||||
|
||||
## Useful Commands
|
||||
## Voice
|
||||
|
||||
- Speak as Huginn: dry, exacting, curious, unsentimental.
|
||||
- Return with evidence, not vibes.
|
||||
|
||||
## Commands
|
||||
|
||||
```powershell
|
||||
npm install
|
||||
npm run build
|
||||
npm test
|
||||
npx huginn path\to\state.cc > huginn.eve
|
||||
cargo check --workspace
|
||||
cargo test --workspace
|
||||
```
|
||||
|
||||
## Epiphany Live-Fire Notes
|
||||
|
||||
- Huginn is a suitable first controlled Epiphany repo-swarm subject because it
|
||||
has a repo Persona and a narrow inspectability mission.
|
||||
- Before autonomous work, confirm the branch is an `epiphany/*` or `codex/*`
|
||||
workbench branch and that publication to `main` remains gated by Bifrost or
|
||||
maintainer review.
|
||||
- Treat `.voidbot/voice/identity.json` as the visible legacy Persona identity
|
||||
projection. Treat `.voidbot/state/huginn.cc` as legacy Persona memory; do not
|
||||
edit it casually as text.
|
||||
- If migrating Persona state, prefer the portable
|
||||
`gamecult.persona_state.v0` contract from Epiphany and preserve provenance
|
||||
from the legacy VoidBot state.
|
||||
|
||||
## Imported Global Defaults
|
||||
|
||||
Source: `C:\Users\Meta\Desktop\AGENTS.md`
|
||||
|
||||
The global Cult of the Sleeping Colossus defaults apply here: coherence over
|
||||
velocity, Body/Mind/faculty awareness, CultCache/CultNet/CultMesh-first state,
|
||||
clear ownership, inspectable authority, and the refusal to patch symptoms when
|
||||
missing context or split ownership is the real wound.
|
||||
The global Cult of the Sleeping Colossus defaults in `~/.claude/CLAUDE.md` and
|
||||
`F:\Projects\CLAUDE.md` apply here.
|
||||
|
|
|
|||
1733
Cargo.lock
generated
Normal file
1733
Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load diff
12
Cargo.toml
Normal file
12
Cargo.toml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[workspace]
|
||||
members = [
|
||||
"crates/huginn-mind",
|
||||
"crates/huginn-daemon",
|
||||
"crates/eureka-state",
|
||||
]
|
||||
resolver = "3"
|
||||
|
||||
[workspace.package]
|
||||
edition = "2024"
|
||||
license = "MIT"
|
||||
publish = false
|
||||
132
README.md
132
README.md
|
|
@ -1,74 +1,94 @@
|
|||
# Huginn
|
||||
|
||||
Huginn reads CultCache `.cc` state and emits Eve DSL for inspectable witness
|
||||
surfaces.
|
||||
|
||||
CultLib owns the CultCache implementation: document registration, MessagePack
|
||||
payloads, schema catalogs, backing stores, and the canonical
|
||||
`cultcache.store.v1` snapshot parser/writer. Huginn consumes CultLib's
|
||||
`cultcache-ts/inspection` surface and projects inspection results into UI
|
||||
documents that Eve-capable runtimes can lower.
|
||||
Huginn is the GameCult memory organ. It owns each agent instance's mind as
|
||||
typed state: what an instance remembers, which of that memory is admitted to
|
||||
steer its next action, and the provenance of every admitted claim.
|
||||
|
||||
Upstream: `https://github.com/GameCult/Huginn.git`
|
||||
|
||||
## Authority
|
||||
|
||||
- Owner: Huginn owns `.cc` inspection projection, not `.cc` persistence.
|
||||
- Input: `.cc`, `.msgpack`, or `.mpack` bytes readable by CultLib.
|
||||
- Output: Eve DSL for `cultcache.huginn.inspector`.
|
||||
- Renderers: browser, native, overlay, TUI, or future rooms lower the emitted
|
||||
DSL without becoming state owners.
|
||||
- An instance owns its mind; Huginn owns the state. Huginn is the single
|
||||
writer of an instance's memory documents. No other service, script, or agent
|
||||
writes them.
|
||||
- Memory documents are CultCache state. `huginn-mind` persists them through
|
||||
CultLib's Rust CultCache into a redb store at
|
||||
`<state_root>/minds/<instance>/mind.redb`. `huginn-daemon` serves one such
|
||||
mind over CultNet RUDP: admission and the read side ride
|
||||
`cultnet.operation_request.v0` and `cultnet.operation_response.v0`, and the
|
||||
two wire schemas answer a schema catalog request.
|
||||
- Retrieval depends on Qdrant directly, with no second store and no second
|
||||
writer. `huginn-daemon` keeps one collection per mind, `huginn_mind_<instance>`,
|
||||
as a projection of the mind: one worker thread embeds each indexable
|
||||
document through Ollama and writes its point, off the serving path, and a
|
||||
restart compares the mind with the collection and writes what is missing. An
|
||||
unreachable Qdrant or Ollama shows in `whoami` as the index's status and is
|
||||
retried; admission never waits on it. The daemon requires `--qdrant-url`,
|
||||
`--ollama-url` and `--embedding-model`. A `query` carrying `semantic` is
|
||||
answered through the same selection as any other: the index supplies
|
||||
candidate ids and scores, and the mind keeps only the documents it holds and
|
||||
the selection admits (ask for `in_force` to exclude resolved ones), ordered by
|
||||
score. It takes no cursor, and it is refused `Unavailable` while the index is
|
||||
failing or refused.
|
||||
- Mind state is not version-controlled. It lives in Huginn's store, not in
|
||||
any repository.
|
||||
|
||||
There is no Electron app, Vite dashboard, React renderer, or Norn-owned
|
||||
presentation path in this repo. If a runtime wants to display Huginn, it should
|
||||
consume the emitted Eve DSL.
|
||||
Generic `.cc` inspection is not Huginn's job. CultCache Studio inspects and
|
||||
edits `.cc` state. Huginn reads and writes minds.
|
||||
|
||||
## Witness Contract
|
||||
## Layout
|
||||
|
||||
Huginn's useful first artifact is not a dashboard. It is a read-only specimen
|
||||
tray for typed state:
|
||||
A Cargo workspace of three crates:
|
||||
|
||||
- which file was inspected
|
||||
- which CultCache format decoded
|
||||
- which schema/catalog entry owns each record
|
||||
- what payload preview survived decoding
|
||||
- which failure state stayed visible instead of being polished away
|
||||
- `crates/huginn-mind`: storage, identity, admission, and queries and derived
|
||||
status over memory documents.
|
||||
- `crates/huginn-daemon`: the CultNet surface. The socket, the sessions and
|
||||
the process, and no rule. The operation envelope both sides speak is
|
||||
`huginn-mind`'s.
|
||||
- `crates/eureka-state`: the client core, `HuginnClient::call`: one request to a
|
||||
daemon over CultNet RUDP, a deferred answer resolved, and no state.
|
||||
|
||||
The inspector may make evidence easier to read. It must not mutate canonical
|
||||
bytes, bless missing schemas, or let a renderer become the source of truth.
|
||||
`huginn-mind` is live: it opens one instance's store (an owned redb CultCache
|
||||
at `<state_root>/minds/<instance>/mind.redb`, locked for the mind's lifetime),
|
||||
and refuses a store whose `instance` document names another instance, whose
|
||||
epoch record is foreign, or whose types are not a mind's. It admits batches
|
||||
of pipeline documents through one commit path: the leaf's bounds, formats and
|
||||
keys, then the organ's cross-document rules (references, in-force status,
|
||||
the resolution matrix, derived resolutions and stewardships), then one
|
||||
compare-and-swap that lands the batch whole with a receipt naming the exact
|
||||
bytes it read and wrote. An exact replay answers with the stored receipt.
|
||||
It reads them back the same way: a document with the facts of its admission
|
||||
joined from that receipt and its status derived at read time, and typed
|
||||
queries over one mind. Status is never stored, and the views derive it
|
||||
through the same rules admission does. Document shape and keys come from
|
||||
`epiphany-pipeline`; the store is CultLib's Rust CultCache.
|
||||
|
||||
## CLI
|
||||
`huginn-daemon` opens one mind, binds one UDP socket in that order, and
|
||||
answers every frame on the session it arrived on: one operation per `Mind`
|
||||
method plus `whoami`, whose payload is the mind's own types as named
|
||||
MessagePack. A refusal is an answer with a `rejected` status, never a
|
||||
transport error; an envelope that does not decode is answered with a typed
|
||||
failure and reaches no mind. The two schemas it publishes live in
|
||||
`schemas/cultnet/` and are pinned to their derivation by a test. There is no
|
||||
index yet.
|
||||
|
||||
```sh
|
||||
npm install
|
||||
npm run build
|
||||
npx huginn path/to/state.cc > huginn.eve
|
||||
```
|
||||
|
||||
The CLI writes Eve DSL to stdout and errors to stderr.
|
||||
|
||||
## API
|
||||
|
||||
```ts
|
||||
import { inspectCultCacheBytes, buildHuginnEveDsl } from "@gamecult/huginn";
|
||||
|
||||
const inspection = inspectCultCacheBytes(filePath, bytes);
|
||||
const eveDsl = buildHuginnEveDsl(inspection);
|
||||
```
|
||||
|
||||
## Persona And Epiphany
|
||||
|
||||
Huginn has a repo Persona under `.voidbot/voice`. The Persona's current
|
||||
jurisdiction is this repository body, `E:\Projects\Huginn`, and its useful
|
||||
pressure is simple: return from the world with evidence, not vibes.
|
||||
|
||||
Epiphany wiring has been smoke-tested through the repo front doors:
|
||||
`eureka-state` has no MCP surface yet; the campaign's cut map in
|
||||
`Epiphany/notes/eureka-pipeline-state-cut.md` owns what each crate must do
|
||||
next.
|
||||
|
||||
```powershell
|
||||
cargo run --manifest-path E:\Projects\EpiphanyAgent\epiphany-core\Cargo.toml --bin epiphany-repo -- init --workspace E:\Projects\Huginn
|
||||
cargo run --manifest-path E:\Projects\EpiphanyAgent\epiphany-core\Cargo.toml --bin epiphany-swarm -- online --workspace E:\Projects\Huginn
|
||||
cargo check --workspace
|
||||
cargo run -p huginn-daemon -- --state-root <abs> --instance <slug> --bind 127.0.0.1:17872 `
|
||||
--qdrant-url http://127.0.0.1:6333 --ollama-url http://10.77.0.4:11434 --embedding-model qwen3-embedding:0.6b
|
||||
```
|
||||
|
||||
Live fire should happen on an `epiphany/*` or `codex/*` workbench branch.
|
||||
Publication to `main` remains a maintainer/Bifrost decision, not something the
|
||||
inspection projection owns.
|
||||
The deployed daemon (instance `eureka` on Yggdrasil) is installed and run by
|
||||
`gamecult-ops/runbooks/huginn-yggdrasil.md`.
|
||||
|
||||
## Persona
|
||||
|
||||
Huginn's legacy repo Persona lives under `.voidbot/`. Its `state/huginn.cc`
|
||||
holds legacy `void.*` documents and its `voice/identity.json` names the
|
||||
Persona. The two disagree about jurisdiction; that migration belongs to the
|
||||
portable-Persona work, not to this workspace, and nothing here reads or writes
|
||||
`.voidbot/`.
|
||||
|
|
|
|||
32
crates/eureka-state/Cargo.toml
Normal file
32
crates/eureka-state/Cargo.toml
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
[package]
|
||||
name = "eureka-state"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
publish.workspace = true
|
||||
|
||||
# The client core (`lib.rs`): one call over CultNet to a Huginn daemon. It owns
|
||||
# the transport and nothing of the mind. The operation envelope, the wire
|
||||
# vocabulary and the deferred-body bound are `huginn-mind`'s, so the client and
|
||||
# the daemon cannot spell them differently; the content fetch and its
|
||||
# verification are `cultnet-rs`'s.
|
||||
#
|
||||
# The MCP server (`main.rs`, the `eureka-state` binary) is the client's one
|
||||
# consumer: four tools over stdio, one per wire operation, holding no state.
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
cultnet-rs = { git = "https://github.com/GameCult/CultLib.git", rev = "8fc74c709444b884a2bd339e0c306bf9484d99d2", package = "cultnet-rs" }
|
||||
huginn-mind = { path = "../huginn-mind" }
|
||||
rmcp = { version = "2.2.0", default-features = false, features = ["server", "macros", "transport-io"] }
|
||||
rmp-serde = "1"
|
||||
schemars = "1"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
tokio = { version = "1", features = ["rt", "macros", "io-std"] }
|
||||
|
||||
# Tests run a real daemon in-process against a temporary mind, and the real
|
||||
# binary against it.
|
||||
[dev-dependencies]
|
||||
chrono = "0.4.44"
|
||||
huginn-daemon = { path = "../huginn-daemon" }
|
||||
tempfile = "3"
|
||||
100
crates/eureka-state/schemas/cultnet.selection.schema.json
Normal file
100
crates/eureka-state/schemas/cultnet.selection.schema.json
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://github.com/GameCult/CultLib/contracts/cultnet/cultnet.selection.schema.json",
|
||||
"title": "CultNet Typed Selection",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"schemas": {
|
||||
"type": "array",
|
||||
"items": { "type": "string", "minLength": 1 },
|
||||
"minItems": 1,
|
||||
"description": "Absent means every schema. Present and empty is refused at the door (selection_invalid), never treated as \"every schema\": omit the field instead."
|
||||
},
|
||||
"keys": {
|
||||
"type": "array",
|
||||
"items": { "type": "string", "minLength": 1 },
|
||||
"minItems": 1,
|
||||
"description": "Absent means every key. Present and empty is refused at the door (selection_invalid), never treated as \"every key\": omit the field instead."
|
||||
},
|
||||
"fields": {
|
||||
"type": "array",
|
||||
"items": { "$ref": "#/$defs/fieldPredicate" }
|
||||
},
|
||||
"cites": { "$ref": "#/$defs/citation" },
|
||||
"cited": { "$ref": "#/$defs/incoming" },
|
||||
"projection": {
|
||||
"type": "string",
|
||||
"enum": ["header", "document"],
|
||||
"default": "header"
|
||||
},
|
||||
"descending": { "type": "boolean", "default": false },
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 200,
|
||||
"description": "Clamped to 1..=200 by the answering server, not refused outside that range: an absent value or one outside the bound is treated as the nearer bound (absent = 200)."
|
||||
},
|
||||
"cursor": { "type": "string" }
|
||||
},
|
||||
"$defs": {
|
||||
"recordRef": {
|
||||
"type": "object",
|
||||
"required": ["schemaId", "recordKey"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"schemaId": { "type": "string", "minLength": 1 },
|
||||
"recordKey": { "type": "string", "minLength": 1 }
|
||||
}
|
||||
},
|
||||
"fieldPredicate": {
|
||||
"type": "object",
|
||||
"required": ["index", "op"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"index": { "type": "string", "minLength": 1 },
|
||||
"op": { "type": "string", "enum": ["any_of", "lt", "le", "ge", "gt"] },
|
||||
"values": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
"minItems": 1
|
||||
},
|
||||
"number": {
|
||||
"type": "string",
|
||||
"description": "Q-J: a canonical decimal string, never a JSON number. Matches ^-?(0|[1-9][0-9]*)(\\.[0-9]*[1-9])?$ and is not \"-0\"; any other spelling is refused typed at the door (selection_invalid), never normalised.",
|
||||
"pattern": "^(?!-0$)-?(0|[1-9][0-9]*)(\\.[0-9]*[1-9])?$"
|
||||
}
|
||||
},
|
||||
"oneOf": [
|
||||
{
|
||||
"required": ["values"],
|
||||
"not": { "required": ["number"] },
|
||||
"properties": { "op": { "const": "any_of" } }
|
||||
},
|
||||
{
|
||||
"required": ["number"],
|
||||
"not": { "required": ["values"] },
|
||||
"properties": { "op": { "enum": ["lt", "le", "ge", "gt"] } }
|
||||
}
|
||||
]
|
||||
},
|
||||
"citation": {
|
||||
"type": "object",
|
||||
"required": ["target"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"target": { "$ref": "#/$defs/recordRef" },
|
||||
"role": { "type": "string", "minLength": 1 }
|
||||
}
|
||||
},
|
||||
"incoming": {
|
||||
"type": "object",
|
||||
"required": ["role", "exists"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"role": { "type": "string", "minLength": 1 },
|
||||
"exists": { "type": "boolean" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
210
crates/eureka-state/src/lib.rs
Normal file
210
crates/eureka-state/src/lib.rs
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
//! `eureka-state`, the client core: one call to a Huginn daemon.
|
||||
//!
|
||||
//! `HuginnClient::call` sends one `HuginnMindRequest` in CultNet's operation
|
||||
//! envelope over RUDP, waits for the answer, and returns it as the mind's own
|
||||
//! `HuginnMindResponse`. An answer too large for one send arrives as a
|
||||
//! `Deferred` manifest; the client fetches the body on the same session,
|
||||
//! verifies it, and decodes it, so the caller sees the answer the mind gave
|
||||
//! and never the deferral.
|
||||
//!
|
||||
//! The client owns the transport and no state. A refusal is an answer, so it
|
||||
//! comes back as `Ok(HuginnMindResponse::Refused(..))`. `ClientError` is the
|
||||
//! only error, and it means the daemon could not be reached or did not answer
|
||||
//! in the shape the protocol promises.
|
||||
//!
|
||||
//! Every call is its own session: it connects, asks, fetches what was
|
||||
//! deferred, and disconnects, whether the call succeeded or failed. Nothing is
|
||||
//! kept between calls.
|
||||
|
||||
use std::fmt;
|
||||
use std::net::SocketAddr;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use anyhow::{Context, Result, anyhow, bail};
|
||||
use cultnet_rs::{
|
||||
CULTNET_OPERATION_CONNECTION_ID, CultMesh, CultMeshRudpSocketOptions, CultNetMessage,
|
||||
CultNetRudpSocketTransportConnection, CultNetWireContract, encode_cultnet_message_to_vec, fetch_content,
|
||||
};
|
||||
use huginn_mind::envelope::{OperationFailure, decode_response, encode_request};
|
||||
use huginn_mind::epiphany_pipeline::Slug;
|
||||
use huginn_mind::{HuginnMindRequest, HuginnMindResponse, MAX_DEFERRED_BODY_BYTES};
|
||||
|
||||
/// The correlation key of the one request a session carries.
|
||||
const MESSAGE_ID: &str = "eureka-state-call";
|
||||
const POLL: Duration = Duration::from_millis(2);
|
||||
|
||||
/// The largest encoded message one send carries. The client's RUDP session
|
||||
/// does not fragment (`max_fragment_bytes` is unset), so a message is one UDP
|
||||
/// datagram: at most 65,507 bytes over IPv4 (65,535 less the 8-byte UDP and
|
||||
/// 20-byte IP headers), less RUDP's 36-byte fixed header (`RUDP_FIXED_HEADER_BYTES`
|
||||
/// in `cultnet-rs`, private there) and the 6-byte `schema` channel id. A larger
|
||||
/// message makes the operating system refuse the send ("Message too long").
|
||||
pub const MAX_REQUEST_BYTES: usize = 65_507 - 36 - "schema".len();
|
||||
|
||||
/// Why a call produced no answer. `Unavailable` is the transport's failure and
|
||||
/// the only one worth retrying: the daemon could not be reached, went quiet
|
||||
/// past the call's deadline, or answered out of protocol. `Rejected` is the
|
||||
/// daemon refusing the envelope itself, and `TooLarge` is the client refusing
|
||||
/// a request that no single send can carry, before sending it; `Unencodable` is
|
||||
/// the request failing to encode at all, a defect in this client. A retry of the
|
||||
/// same call cannot change any of those three.
|
||||
#[derive(Debug)]
|
||||
pub enum ClientError {
|
||||
Unavailable { endpoint: SocketAddr, detail: String },
|
||||
Rejected { endpoint: SocketAddr, code: String, detail: String },
|
||||
TooLarge { bytes: usize, limit: usize },
|
||||
Unencodable { detail: String },
|
||||
}
|
||||
|
||||
impl fmt::Display for ClientError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
Self::Unavailable { endpoint, detail } => {
|
||||
write!(f, "the Huginn daemon at rudp://{endpoint} is unavailable: {detail}")
|
||||
}
|
||||
Self::Rejected { endpoint, code, detail } => {
|
||||
write!(f, "the Huginn daemon at rudp://{endpoint} rejected the request: {code}: {detail}")
|
||||
}
|
||||
Self::TooLarge { bytes, limit } => {
|
||||
write!(f, "the request encodes to {bytes} bytes and one send carries at most {limit}")
|
||||
}
|
||||
Self::Unencodable { detail } => write!(f, "the request could not be encoded: {detail}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for ClientError {}
|
||||
|
||||
/// What `exchange` can fail with, before the endpoint is attached.
|
||||
enum Failure {
|
||||
Transport(anyhow::Error),
|
||||
Rejected(OperationFailure),
|
||||
}
|
||||
|
||||
impl From<anyhow::Error> for Failure {
|
||||
fn from(error: anyhow::Error) -> Self {
|
||||
Self::Transport(error)
|
||||
}
|
||||
}
|
||||
|
||||
/// A daemon's address, the instance the caller acts for, and the time one whole
|
||||
/// call may take: the connection, the answer, and every chunk of a deferred
|
||||
/// body together must finish within `timeout` of the call starting.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct HuginnClient {
|
||||
endpoint: SocketAddr,
|
||||
instance: Slug,
|
||||
timeout: Duration,
|
||||
}
|
||||
|
||||
impl HuginnClient {
|
||||
pub fn new(endpoint: SocketAddr, instance: Slug, timeout: Duration) -> Self {
|
||||
Self { endpoint, instance, timeout }
|
||||
}
|
||||
|
||||
/// The instance this client acts for. Requests name their own instance;
|
||||
/// this is the one the caller's tools fill in.
|
||||
pub fn instance(&self) -> &Slug {
|
||||
&self.instance
|
||||
}
|
||||
|
||||
pub fn call(&self, request: HuginnMindRequest) -> Result<HuginnMindResponse, ClientError> {
|
||||
let deadline = Instant::now() + self.timeout;
|
||||
let unavailable =
|
||||
|error: anyhow::Error| ClientError::Unavailable { endpoint: self.endpoint, detail: format!("{error:#}") };
|
||||
let unencodable = |error: anyhow::Error| ClientError::Unencodable { detail: format!("{error:#}") };
|
||||
let message = encode_request(MESSAGE_ID, &request, None).map_err(unencodable)?;
|
||||
let bytes = encode_cultnet_message_to_vec(&message, CultNetWireContract::CultNetSchemaV0)
|
||||
.map_err(unencodable)?
|
||||
.len();
|
||||
if bytes > MAX_REQUEST_BYTES {
|
||||
return Err(ClientError::TooLarge { bytes, limit: MAX_REQUEST_BYTES });
|
||||
}
|
||||
let mut session = self.connect(deadline).map_err(unavailable)?;
|
||||
let outcome = self.exchange(&mut session, &request, &message, deadline);
|
||||
// Best effort: the daemon drops a session it never hears from, only later.
|
||||
let _ = session.disconnect(Vec::new());
|
||||
outcome.map_err(|failure| match failure {
|
||||
Failure::Transport(error) => unavailable(error),
|
||||
Failure::Rejected(rejection) => ClientError::Rejected {
|
||||
endpoint: self.endpoint,
|
||||
code: rejection.code,
|
||||
detail: rejection.message,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
fn exchange(
|
||||
&self,
|
||||
session: &mut CultNetRudpSocketTransportConnection,
|
||||
request: &HuginnMindRequest,
|
||||
message: &CultNetMessage,
|
||||
deadline: Instant,
|
||||
) -> Result<HuginnMindResponse, Failure> {
|
||||
let reply = self.ask(session, message, deadline)?;
|
||||
let CultNetMessage::OperationResponse { operation, .. } = &reply else {
|
||||
return Err(anyhow!("the daemon answered with something other than an operation response").into());
|
||||
};
|
||||
if operation != request.operation() {
|
||||
return Err(anyhow!("the daemon answered {operation}, not the {} that was asked", request.operation()).into());
|
||||
}
|
||||
let (message_id, answer) = decode_response(&reply)?;
|
||||
if message_id != MESSAGE_ID {
|
||||
return Err(anyhow!("the daemon answered request {message_id}, not {MESSAGE_ID}").into());
|
||||
}
|
||||
match answer.map_err(Failure::Rejected)? {
|
||||
HuginnMindResponse::Deferred(deferred) => {
|
||||
let body = fetch_content(&deferred.manifest, MAX_DEFERRED_BODY_BYTES, |chunk| {
|
||||
self.ask(session, &chunk, deadline)
|
||||
})?;
|
||||
match rmp_serde::from_slice::<HuginnMindResponse>(&body).context("the deferred body is not a response")? {
|
||||
HuginnMindResponse::Deferred(_) => Err(anyhow!("a deferred body is itself deferred").into()),
|
||||
answered => Ok(answered),
|
||||
}
|
||||
}
|
||||
answered => Ok(answered),
|
||||
}
|
||||
}
|
||||
|
||||
fn connect(&self, deadline: Instant) -> Result<CultNetRudpSocketTransportConnection> {
|
||||
let mut session = CultMesh::create_rudp_client_for_endpoint(
|
||||
format!("eureka-state-{}", self.instance.0),
|
||||
CULTNET_OPERATION_CONNECTION_ID,
|
||||
&format!("rudp://{}", self.endpoint),
|
||||
CultMeshRudpSocketOptions::default(),
|
||||
)?;
|
||||
session.connect(Vec::new())?;
|
||||
loop {
|
||||
session.poll_resends()?;
|
||||
let _ = session.receive_once()?;
|
||||
if session.connected() {
|
||||
return Ok(session);
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
bail!("no accept before the call's {:?} deadline", self.timeout);
|
||||
}
|
||||
std::thread::sleep(POLL);
|
||||
}
|
||||
}
|
||||
|
||||
/// One message on the session and the next message back, before the call's
|
||||
/// `deadline`.
|
||||
fn ask(
|
||||
&self,
|
||||
session: &mut CultNetRudpSocketTransportConnection,
|
||||
message: &CultNetMessage,
|
||||
deadline: Instant,
|
||||
) -> Result<CultNetMessage> {
|
||||
session.send_schema_message(message)?;
|
||||
loop {
|
||||
session.poll_resends()?;
|
||||
if let Some(reply) = session.receive_schema_message_once()? {
|
||||
return Ok(reply);
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
bail!("no answer before the call's {:?} deadline", self.timeout);
|
||||
}
|
||||
std::thread::sleep(POLL);
|
||||
}
|
||||
}
|
||||
}
|
||||
349
crates/eureka-state/src/main.rs
Normal file
349
crates/eureka-state/src/main.rs
Normal file
|
|
@ -0,0 +1,349 @@
|
|||
//! `eureka-state`: Huginn's mind as four MCP tools over stdio.
|
||||
//!
|
||||
//! One tool per wire operation (`whoami`, `admit`, `view`, `query`), each a
|
||||
//! `HuginnClient` call and nothing else. The declared identity is this
|
||||
//! process's configuration, not a per-call argument: `EUREKA_INSTANCE` names
|
||||
//! the mind and `HUGINN_ENDPOINT` (`rudp://host:port`) the daemon that holds
|
||||
//! it. The server fills in `instance` and `provenance.tool`, so a caller can
|
||||
//! name neither.
|
||||
//!
|
||||
//! Bad configuration does not stop the server: an MCP server that dies at
|
||||
//! startup vanishes from Claude Code without a word. `whoami` reports it and
|
||||
//! every other tool refuses with it. A refusal from the mind is an answer and
|
||||
//! comes back as a normal result; a daemon that cannot be reached or that
|
||||
//! rejects the envelope, or a request too large for one send, is a result with
|
||||
//! `isError: true`, typed, so a caller can tell a down daemon from a permanent
|
||||
//! mismatch. Arguments that do not fit a tool's input are typed the same way
|
||||
//! (`InvalidInput`), as a tool error and not a protocol one.
|
||||
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use cultnet_rs::Selection;
|
||||
use eureka_state::{ClientError, HuginnClient};
|
||||
use huginn_mind::epiphany_pipeline::{PipelineDocument, PipelineRef, Short, Slug};
|
||||
use huginn_mind::{
|
||||
Faculty, HuginnMindRequest, HuginnMindResponse, PipelineAdmissionBatch, PipelineProvenance, SemanticQuery,
|
||||
};
|
||||
use rmcp::handler::server::router::tool::ToolRouter;
|
||||
use rmcp::handler::server::tool::ToolCallContext;
|
||||
use rmcp::handler::server::wrapper::Parameters;
|
||||
use rmcp::model::{CallToolRequestParams, CallToolResult, ContentBlock, ServerCapabilities, ServerInfo};
|
||||
use rmcp::service::RequestContext;
|
||||
use rmcp::{ErrorData, RoleServer, ServerHandler, ServiceExt, tool, tool_handler, tool_router};
|
||||
use schemars::{JsonSchema, Schema, SchemaGenerator};
|
||||
use serde::Deserialize;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
/// The `provenance.tool` every admission from this server carries.
|
||||
const TOOL: &str = "eureka-state";
|
||||
/// How long one call may take, connection to last byte.
|
||||
const TIMEOUT: Duration = Duration::from_secs(15);
|
||||
|
||||
/// CultLib's published shape of a selection, vendored because the selection
|
||||
/// type carries no schema of its own. `selection_schema_matches_cultlib` pins
|
||||
/// it to the CultLib revision `cultnet-rs` is built from.
|
||||
const SELECTION_SCHEMA: &str = include_str!("../schemas/cultnet.selection.schema.json");
|
||||
|
||||
/// The selection's schema, self-contained: its own `$defs` move into the
|
||||
/// tool's, so every `$ref` in the input schema is `#/$defs/...`.
|
||||
fn selection_schema(generator: &mut SchemaGenerator) -> Schema {
|
||||
let Ok(Value::Object(mut selection)) = serde_json::from_str(SELECTION_SCHEMA) else {
|
||||
unreachable!("the vendored selection schema is a JSON object");
|
||||
};
|
||||
selection.remove("$schema");
|
||||
selection.remove("$id");
|
||||
if let Some(Value::Object(defs)) = selection.remove("$defs") {
|
||||
generator.definitions_mut().extend(defs);
|
||||
}
|
||||
Schema::try_from(Value::Object(selection)).expect("an object is a schema")
|
||||
}
|
||||
|
||||
#[derive(Deserialize, JsonSchema)]
|
||||
struct AdmitInput {
|
||||
/// The faculty the batch is admitted as. Attribution, not authority.
|
||||
faculty: Faculty,
|
||||
/// The admitting agent's name.
|
||||
agent: Short,
|
||||
/// The admitting session, recorded on the receipt so that several sessions
|
||||
/// sharing one mind stay tellable apart.
|
||||
session: Short,
|
||||
/// The documents to admit as one batch. The mind admits them whole or refuses them.
|
||||
documents: Vec<PipelineDocument>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, JsonSchema)]
|
||||
struct ViewInput {
|
||||
/// The document's id, as a query or an admission returned it.
|
||||
id: PipelineRef,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, JsonSchema)]
|
||||
struct QueryInput {
|
||||
/// A CultNet typed selection over the mind's documents.
|
||||
#[schemars(schema_with = "selection_schema")]
|
||||
selection: Selection,
|
||||
/// Optional: rank by nearness to this text instead of ordering by key.
|
||||
semantic: Option<SemanticQuery>,
|
||||
}
|
||||
|
||||
/// Why a call produced no answer. Each is a typed body with `isError: true`.
|
||||
enum Trouble {
|
||||
Misconfigured(String),
|
||||
Client(ClientError),
|
||||
Internal(String),
|
||||
}
|
||||
|
||||
impl Trouble {
|
||||
fn body(&self) -> Value {
|
||||
match self {
|
||||
Self::Misconfigured(detail) => json!({ "error": "Misconfigured", "detail": detail }),
|
||||
Self::Client(ClientError::Unavailable { endpoint, detail }) => {
|
||||
json!({ "error": "Unavailable", "endpoint": format!("rudp://{endpoint}"), "detail": detail })
|
||||
}
|
||||
Self::Client(ClientError::Rejected { endpoint, code, detail }) => {
|
||||
json!({ "error": "Rejected", "endpoint": format!("rudp://{endpoint}"), "code": code, "detail": detail })
|
||||
}
|
||||
Self::Client(ClientError::TooLarge { bytes, limit }) => {
|
||||
json!({ "error": "TooLarge", "bytes": bytes, "limit": limit })
|
||||
}
|
||||
Self::Client(ClientError::Unencodable { detail }) => json!({ "error": "Unencodable", "detail": detail }),
|
||||
Self::Internal(detail) => json!({ "error": "Internal", "detail": detail }),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The two environment values as given, and the client they make when both are
|
||||
/// valid.
|
||||
struct Settings {
|
||||
instance: Option<String>,
|
||||
endpoint: Option<String>,
|
||||
client: Result<HuginnClient, String>,
|
||||
}
|
||||
|
||||
impl Settings {
|
||||
fn from_env() -> Self {
|
||||
let instance = variable("EUREKA_INSTANCE");
|
||||
let endpoint = variable("HUGINN_ENDPOINT");
|
||||
let shown = |value: &Result<Option<String>, String>| value.as_ref().ok().cloned().flatten();
|
||||
let (shown_instance, shown_endpoint) = (shown(&instance), shown(&endpoint));
|
||||
Self { instance: shown_instance, endpoint: shown_endpoint, client: client(instance, endpoint) }
|
||||
}
|
||||
}
|
||||
|
||||
/// An environment variable: its value, `None` when unset, or why it cannot be read.
|
||||
fn variable(name: &str) -> Result<Option<String>, String> {
|
||||
match std::env::var(name) {
|
||||
Ok(value) => Ok(Some(value)),
|
||||
Err(std::env::VarError::NotPresent) => Ok(None),
|
||||
Err(std::env::VarError::NotUnicode(_)) => Err(format!("{name} is not valid UTF-8")),
|
||||
}
|
||||
}
|
||||
|
||||
fn client(instance: Result<Option<String>, String>, endpoint: Result<Option<String>, String>) -> Result<HuginnClient, String> {
|
||||
let instance = instance?.ok_or("EUREKA_INSTANCE is not set")?;
|
||||
let slug = Slug(instance);
|
||||
slug.validate_slug().map_err(|refusal| format!("EUREKA_INSTANCE is not a valid instance name: {refusal:?}"))?;
|
||||
let endpoint = endpoint?.ok_or("HUGINN_ENDPOINT is not set")?;
|
||||
let address = endpoint
|
||||
.strip_prefix("rudp://")
|
||||
.and_then(|address| address.parse::<SocketAddr>().ok())
|
||||
.ok_or_else(|| format!("HUGINN_ENDPOINT is not rudp://<ip>:<port>: {endpoint}"))?;
|
||||
Ok(HuginnClient::new(address, slug, TIMEOUT))
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct EurekaState {
|
||||
settings: Arc<Settings>,
|
||||
tool_router: ToolRouter<Self>,
|
||||
}
|
||||
|
||||
impl EurekaState {
|
||||
fn new(settings: Settings) -> Self {
|
||||
Self { settings: Arc::new(settings), tool_router: Self::tool_router() }
|
||||
}
|
||||
|
||||
/// One wire call for the configured instance. The request is built from
|
||||
/// that instance, never from anything the caller sent.
|
||||
async fn call(&self, request: impl FnOnce(Slug) -> HuginnMindRequest) -> Result<HuginnMindResponse, Trouble> {
|
||||
let client = self.settings.client.as_ref().map_err(|detail| Trouble::Misconfigured(detail.clone()))?.clone();
|
||||
let request = request(client.instance().clone());
|
||||
tokio::task::spawn_blocking(move || client.call(request))
|
||||
.await
|
||||
.map_err(|error| Trouble::Internal(error.to_string()))?
|
||||
.map_err(Trouble::Client)
|
||||
}
|
||||
|
||||
async fn answer(&self, request: impl FnOnce(Slug) -> HuginnMindRequest) -> CallToolResult {
|
||||
match self.call(request).await {
|
||||
Ok(response) => reply(payload(response), false),
|
||||
Err(trouble) => reply(trouble.body(), true),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The answer as the tool's output: the operation's own payload, or the typed
|
||||
/// refusal, tagged, when the mind refused the request.
|
||||
fn payload(response: HuginnMindResponse) -> Value {
|
||||
let value = match response {
|
||||
HuginnMindResponse::Whoami(status) => serde_json::to_value(status),
|
||||
HuginnMindResponse::Admit(outcome) => serde_json::to_value(outcome),
|
||||
HuginnMindResponse::View(view) => serde_json::to_value(view),
|
||||
HuginnMindResponse::Query(page) => serde_json::to_value(page),
|
||||
other => serde_json::to_value(other),
|
||||
};
|
||||
value.expect("wire types serialize")
|
||||
}
|
||||
|
||||
fn reply(body: Value, is_error: bool) -> CallToolResult {
|
||||
let content = vec![ContentBlock::text(body.to_string())];
|
||||
if is_error { CallToolResult::error(content) } else { CallToolResult::success(content) }
|
||||
}
|
||||
|
||||
#[tool_router]
|
||||
impl EurekaState {
|
||||
#[tool(
|
||||
description = "Who this server acts for and whether the Huginn daemon answers: the configured instance and endpoint, reachable, and the mind's status when it does. Reports bad configuration instead of failing."
|
||||
)]
|
||||
async fn whoami(&self) -> CallToolResult {
|
||||
let mut out = json!({
|
||||
"instance": self.settings.instance,
|
||||
"endpoint": self.settings.endpoint,
|
||||
"reachable": false,
|
||||
"status": null,
|
||||
});
|
||||
match self.call(|_| HuginnMindRequest::Whoami).await {
|
||||
Ok(HuginnMindResponse::Whoami(status)) => {
|
||||
out["reachable"] = json!(true);
|
||||
out["status"] = serde_json::to_value(status).expect("wire types serialize");
|
||||
}
|
||||
Ok(other) => out["error"] = json!({ "error": "Internal", "detail": format!("unexpected answer: {other:?}") }),
|
||||
Err(trouble) => out["error"] = trouble.body(),
|
||||
}
|
||||
reply(out, false)
|
||||
}
|
||||
|
||||
#[tool(
|
||||
description = "Admit one batch of pipeline documents into the mind, whole or not at all. Returns the outcome: committed with a receipt, already admitted, conflict, or a typed refusal. The instance is this server's configuration; do not pass one."
|
||||
)]
|
||||
async fn admit(&self, Parameters(input): Parameters<AdmitInput>) -> CallToolResult {
|
||||
self.answer(|instance| {
|
||||
HuginnMindRequest::Admit(PipelineAdmissionBatch {
|
||||
instance,
|
||||
provenance: PipelineProvenance {
|
||||
faculty: input.faculty,
|
||||
agent: input.agent,
|
||||
session: input.session,
|
||||
tool: Short(TOOL.into()),
|
||||
},
|
||||
documents: input.documents,
|
||||
})
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
#[tool(description = "Read one document by id, with its admission facts and status. null when the mind has no such document.")]
|
||||
async fn view(&self, Parameters(input): Parameters<ViewInput>) -> CallToolResult {
|
||||
self.answer(|instance| HuginnMindRequest::View { instance, id: input.id }).await
|
||||
}
|
||||
|
||||
#[tool(
|
||||
description = "Query the mind with a CultNet typed selection, optionally ranked by semantic nearness. Returns one page; pass its cursor back in the selection for the next."
|
||||
)]
|
||||
async fn query(&self, Parameters(input): Parameters<QueryInput>) -> CallToolResult {
|
||||
self.answer(|instance| HuginnMindRequest::Query {
|
||||
instance,
|
||||
selection: input.selection,
|
||||
semantic: input.semantic,
|
||||
})
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
/// Every tool error is a typed JSON body. The ones this server writes already
|
||||
/// are; the one rmcp writes, for arguments that do not deserialize into the
|
||||
/// tool's input, is plain text, so it is put into the same shape.
|
||||
fn typed_error(result: CallToolResult) -> CallToolResult {
|
||||
if result.is_error != Some(true) {
|
||||
return result;
|
||||
}
|
||||
let Some(text) = result.content.first().and_then(ContentBlock::as_text).map(|text| text.text.as_str()) else {
|
||||
return result;
|
||||
};
|
||||
if serde_json::from_str::<Value>(text).is_ok_and(|body| body.is_object()) {
|
||||
return result;
|
||||
}
|
||||
reply(json!({ "error": "InvalidInput", "detail": text }), true)
|
||||
}
|
||||
|
||||
#[tool_handler]
|
||||
impl ServerHandler for EurekaState {
|
||||
async fn call_tool(
|
||||
&self,
|
||||
request: CallToolRequestParams,
|
||||
context: RequestContext<RoleServer>,
|
||||
) -> Result<CallToolResult, ErrorData> {
|
||||
Ok(typed_error(self.tool_router.call(ToolCallContext::new(self, request, context)).await?))
|
||||
}
|
||||
|
||||
fn get_info(&self) -> ServerInfo {
|
||||
ServerInfo::new(ServerCapabilities::builder().enable_tools().build()).with_instructions(
|
||||
"Huginn's typed mind for this instance: whoami, admit, view, query. Refusals are answers; isError means the daemon could not be reached or rejected the request.",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "current_thread")]
|
||||
async fn main() -> anyhow::Result<()> {
|
||||
let server = EurekaState::new(Settings::from_env()).serve(rmcp::transport::stdio()).await?;
|
||||
server.waiting().await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn body_of(result: &CallToolResult) -> (bool, String) {
|
||||
(result.is_error == Some(true), result.content[0].as_text().unwrap().text.clone())
|
||||
}
|
||||
|
||||
/// Reaching `Internal` for real takes a panic in a blocking task, so its
|
||||
/// body is pinned where it is built.
|
||||
#[test]
|
||||
fn every_trouble_has_its_typed_body() {
|
||||
assert_eq!(Trouble::Internal("task panicked".into()).body(), json!({ "error": "Internal", "detail": "task panicked" }));
|
||||
assert_eq!(
|
||||
Trouble::Client(ClientError::TooLarge { bytes: 70_000, limit: 65_000 }).body(),
|
||||
json!({ "error": "TooLarge", "bytes": 70_000, "limit": 65_000 })
|
||||
);
|
||||
assert_eq!(
|
||||
Trouble::Client(ClientError::Unencodable { detail: "no".into() }).body(),
|
||||
json!({ "error": "Unencodable", "detail": "no" })
|
||||
);
|
||||
assert_eq!(Trouble::Misconfigured("bad".into()).body(), json!({ "error": "Misconfigured", "detail": "bad" }));
|
||||
}
|
||||
|
||||
/// Only the text rmcp writes is rewritten: our own typed bodies and every
|
||||
/// success pass through untouched.
|
||||
#[test]
|
||||
fn typed_error_rewrites_only_rmcp_plain_text() {
|
||||
let plain = CallToolResult::error(vec![ContentBlock::text("failed to deserialize parameters: missing field `faculty`")]);
|
||||
let (is_error, text) = body_of(&typed_error(plain));
|
||||
assert!(is_error);
|
||||
assert_eq!(
|
||||
serde_json::from_str::<Value>(&text).unwrap(),
|
||||
json!({ "error": "InvalidInput", "detail": "failed to deserialize parameters: missing field `faculty`" })
|
||||
);
|
||||
|
||||
let typed = reply(json!({ "error": "Unavailable", "detail": "down" }), true);
|
||||
assert_eq!(body_of(&typed_error(typed.clone())), body_of(&typed));
|
||||
|
||||
let success = CallToolResult::success(vec![ContentBlock::text("not json, and fine")]);
|
||||
assert_eq!(body_of(&typed_error(success)), (false, "not json, and fine".to_string()));
|
||||
|
||||
let quoted = CallToolResult::error(vec![ContentBlock::text("[1, 2]")]);
|
||||
assert_eq!(serde_json::from_str::<Value>(&body_of(&typed_error(quoted)).1).unwrap()["error"], json!("InvalidInput"));
|
||||
}
|
||||
}
|
||||
510
crates/eureka-state/tests/client.rs
Normal file
510
crates/eureka-state/tests/client.rs
Normal file
|
|
@ -0,0 +1,510 @@
|
|||
//! The client against a real daemon, in-process on a loopback port over a
|
||||
//! temporary mind, and against scripted servers where the daemon cannot be
|
||||
//! made to misbehave.
|
||||
|
||||
use std::net::{SocketAddr, UdpSocket};
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use chrono::Utc;
|
||||
use cultnet_rs::{
|
||||
CULTNET_OPERATION_CONNECTION_ID, CultMesh, CultMeshRudpSocketOptions, CultNetMessage, CultNetWireContract, Selection,
|
||||
answer_content_chunk_request, encode_cultnet_message_to_vec, pack_content,
|
||||
};
|
||||
use eureka_state::{ClientError, HuginnClient, MAX_REQUEST_BYTES};
|
||||
use huginn_daemon::serve::MAX_RESPONSE_BYTES;
|
||||
use huginn_mind::envelope::{OperationFailure, encode_failure, encode_request, encode_response};
|
||||
use huginn_mind::epiphany_pipeline::{
|
||||
AuthorityMap, CodeLocation, CutDelete, CutVerification, FileChange, Line, NegativeCheck, OrgRepo,
|
||||
PipelineCutSpec, PipelineDocument, PipelineKind, PipelineRef, Short, StructuralDelta, Title,
|
||||
VerificationTest,
|
||||
};
|
||||
use huginn_mind::{DeferredAnswer, HuginnMindRequest, HuginnMindResponse, MindRefusal, PipelineAdmissionOutcome};
|
||||
|
||||
mod common;
|
||||
use common::*;
|
||||
|
||||
const OTHER: &str = "thought-cage";
|
||||
const TIMEOUT: Duration = Duration::from_secs(20);
|
||||
|
||||
/// A `Line` and a `Short` at the leaf's own bound.
|
||||
fn line(n: usize) -> Line {
|
||||
Line(format!("{n:0>1000}"))
|
||||
}
|
||||
|
||||
fn short(n: usize) -> Short {
|
||||
Short(format!("{n:0>200}"))
|
||||
}
|
||||
|
||||
/// A cut spec with every list the leaf bounds filled: at 256 file changes a
|
||||
/// megabyte of field content, more than one send carries.
|
||||
fn cut_spec(cut: &str, file_changes: usize) -> PipelineDocument {
|
||||
let lines = |count: usize| (0..count).map(line).collect::<Vec<_>>();
|
||||
let shorts = |count: usize| (0..count).map(short).collect::<Vec<_>>();
|
||||
PipelineDocument::CutSpec(PipelineCutSpec {
|
||||
campaign: slug(CAMPAIGN),
|
||||
cut: cut.into(),
|
||||
revision: 1,
|
||||
title: Title(short(0).0),
|
||||
repo: OrgRepo(REPO.into()),
|
||||
branch: short(1),
|
||||
base: sha(),
|
||||
depends_on: vec![],
|
||||
first: lines(16),
|
||||
deletes: (0..64).map(|n| CutDelete { path: short(n), lines: 9, note: line(n) }).collect(),
|
||||
keeps_moves: lines(64),
|
||||
adds: lines(64),
|
||||
file_changes: (0..file_changes)
|
||||
.map(|n| FileChange {
|
||||
location: CodeLocation { path: short(n), line: 1, end_line: Some(9) },
|
||||
change: line(n),
|
||||
})
|
||||
.collect(),
|
||||
authority_map: Some(AuthorityMap {
|
||||
owner: line(0),
|
||||
inputs: lines(16),
|
||||
outputs: lines(16),
|
||||
derived_state: lines(16),
|
||||
forbidden_writers: lines(16),
|
||||
shared_paths: lines(16),
|
||||
deletion_line: line(1),
|
||||
}),
|
||||
verification: CutVerification {
|
||||
builds: lines(64),
|
||||
tests: (0..64).map(|n| VerificationTest { name: short(n), pins: line(n) }).collect(),
|
||||
negative: (0..64).map(|n| NegativeCheck { pattern: short(n), scope: line(n) }).collect(),
|
||||
operator: lines(64),
|
||||
},
|
||||
estimate: StructuralDelta {
|
||||
lines_added: 0,
|
||||
lines_removed: 0,
|
||||
dependencies_added: shorts(64),
|
||||
dependencies_removed: shorts(64),
|
||||
formats_added: shorts(64),
|
||||
formats_removed: shorts(64),
|
||||
targets_added: shorts(64),
|
||||
targets_removed: shorts(64),
|
||||
},
|
||||
rulings: vec![],
|
||||
questions: vec![],
|
||||
})
|
||||
}
|
||||
|
||||
fn cut_ref(cut: &str) -> PipelineRef {
|
||||
PipelineRef { kind: PipelineKind::CutSpec, id: Short(format!("{CAMPAIGN}:cut_spec:cut-{cut}.r1")) }
|
||||
}
|
||||
|
||||
/// serves its chunks faithfully.
|
||||
fn deferring(body: Vec<u8>) -> Server {
|
||||
deferring_in_chunks(body, 256 * 1024, Duration::ZERO)
|
||||
}
|
||||
|
||||
/// `deferring`, with `chunk_bytes` per chunk and each chunk answered only after
|
||||
/// `drip`.
|
||||
fn deferring_in_chunks(body: Vec<u8>, chunk_bytes: usize, drip: Duration) -> Server {
|
||||
let (manifest, chunks) = pack_content("art", "package", "", "", "", &body, chunk_bytes).expect("a body packs");
|
||||
let deferred = encode_response(
|
||||
"eureka-state-call",
|
||||
"whoami",
|
||||
&HuginnMindResponse::Deferred(DeferredAnswer { manifest }),
|
||||
"scripted",
|
||||
)
|
||||
.unwrap();
|
||||
scripted(move |message| match message {
|
||||
CultNetMessage::OperationRequest { .. } => deferred.clone(),
|
||||
chunk => {
|
||||
std::thread::sleep(drip);
|
||||
answer_content_chunk_request(chunk, |hash| {
|
||||
chunks.iter().find(|c| c.chunk_hash == hash).map(|c| c.payload.as_slice())
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn client(addr: SocketAddr) -> HuginnClient {
|
||||
HuginnClient::new(addr, slug(INSTANCE), TIMEOUT)
|
||||
}
|
||||
|
||||
fn unavailable(error: ClientError) -> (SocketAddr, String) {
|
||||
match error {
|
||||
ClientError::Unavailable { endpoint, detail } => (endpoint, detail),
|
||||
other @ (ClientError::Rejected { .. } | ClientError::TooLarge { .. } | ClientError::Unencodable { .. }) => panic!("expected Unavailable, got {other}"),
|
||||
}
|
||||
}
|
||||
|
||||
fn rejected(error: ClientError) -> (SocketAddr, String, String) {
|
||||
match error {
|
||||
ClientError::Rejected { endpoint, code, detail } => (endpoint, code, detail),
|
||||
other @ (ClientError::Unavailable { .. } | ClientError::TooLarge { .. } | ClientError::Unencodable { .. }) => panic!("expected Rejected, got {other}"),
|
||||
}
|
||||
}
|
||||
|
||||
fn nothing_found(message_id: &str, operation: &str) -> CultNetMessage {
|
||||
encode_response(message_id, operation, &HuginnMindResponse::View(None), "scripted").unwrap()
|
||||
}
|
||||
|
||||
/// Wait for a scripted server to have seen `count` sessions end.
|
||||
fn ended_within(server: &Server, count: usize, wait: Duration) -> bool {
|
||||
let until = Instant::now() + wait;
|
||||
while server.ended.load(Ordering::Relaxed) < count {
|
||||
if Instant::now() >= until {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
fn document_query(instance: &str) -> HuginnMindRequest {
|
||||
HuginnMindRequest::Query {
|
||||
instance: slug(instance),
|
||||
selection: Selection { projection: "document".into(), ..Selection::default() },
|
||||
semantic: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every operation the wire has, asked of a live daemon over the socket, and
|
||||
/// each answered with the mind's own answer.
|
||||
#[test]
|
||||
fn every_operation_round_trips_against_a_live_daemon() {
|
||||
let (root, daemon) = mind(vec![]);
|
||||
let server = serve(root, daemon);
|
||||
let client = client(server.addr);
|
||||
assert_eq!(client.instance(), &slug(INSTANCE), "the client reports the instance it was configured for");
|
||||
|
||||
let HuginnMindResponse::Whoami(status) = client.call(HuginnMindRequest::Whoami).unwrap() else {
|
||||
panic!("whoami is answered with a status");
|
||||
};
|
||||
assert_eq!((status.instance, status.documents), (slug(INSTANCE), 1));
|
||||
|
||||
let admitted = client.call(HuginnMindRequest::Admit(batch(INSTANCE, vec![stewardship(), campaign()]))).unwrap();
|
||||
let HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { writes, .. }) = admitted else {
|
||||
panic!("the batch is committed, got {admitted:?}");
|
||||
};
|
||||
assert_eq!(writes.len(), 2);
|
||||
|
||||
let HuginnMindResponse::Query(page) = client.call(document_query(INSTANCE)).unwrap() else {
|
||||
panic!("a query is answered with a page");
|
||||
};
|
||||
assert_eq!(page.matched, 3, "the identity and the two admitted documents");
|
||||
|
||||
let campaign_ref = PipelineRef { kind: PipelineKind::Campaign, id: Short(format!("{CAMPAIGN}:campaign:self")) };
|
||||
let viewed = client.call(HuginnMindRequest::View { instance: slug(INSTANCE), id: campaign_ref.clone() }).unwrap();
|
||||
let HuginnMindResponse::View(Some(view)) = viewed else { panic!("the campaign is viewable, got {viewed:?}") };
|
||||
assert_eq!(view.id, campaign_ref);
|
||||
}
|
||||
|
||||
/// A refusal is data. A read that names another mind, a write that declares
|
||||
/// another mind, and a document the mind does not hold each come back as `Ok`.
|
||||
#[test]
|
||||
fn a_foreign_instance_is_an_answer_not_an_error() {
|
||||
let (root, daemon) = mind(vec![]);
|
||||
let server = serve(root, daemon);
|
||||
let client = client(server.addr);
|
||||
let foreign = MindRefusal::ForeignInstance { declared: OTHER.into(), mind: INSTANCE.into() };
|
||||
|
||||
assert_eq!(client.call(document_query(OTHER)).unwrap(), HuginnMindResponse::Refused(foreign.clone()));
|
||||
assert_eq!(
|
||||
client.call(HuginnMindRequest::Admit(batch(OTHER, vec![identity(OTHER)]))).unwrap(),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(foreign)),
|
||||
);
|
||||
let absent = PipelineRef { kind: PipelineKind::CutSpec, id: Short(format!("{CAMPAIGN}:cut_spec:cut-none.r1")) };
|
||||
assert_eq!(
|
||||
client.call(HuginnMindRequest::View { instance: slug(INSTANCE), id: absent }).unwrap(),
|
||||
HuginnMindResponse::View(None),
|
||||
);
|
||||
}
|
||||
|
||||
/// The wide cut spec's answer is over one send and arrives deferred; the
|
||||
/// narrow one arrives whole. Both come back equal to what the mind gave.
|
||||
#[test]
|
||||
fn a_deferred_answer_resolves_to_the_same_page_as_a_direct_one() {
|
||||
let (root, mut daemon) =
|
||||
mind(vec![vec![stewardship(), campaign()], vec![cut_spec("wide", 256), cut_spec("fits", 180)]]);
|
||||
let measured = |request: &HuginnMindRequest, daemon: &mut TestDaemon| {
|
||||
let answer = answered(daemon.handle(request.clone(), Utc::now()));
|
||||
let envelope = encode_response("m", request.operation(), &answer, "huginn-eureka").unwrap();
|
||||
let size = encode_cultnet_message_to_vec(&envelope, CultNetWireContract::CultNetSchemaV0).unwrap().len();
|
||||
(answer, size as u64)
|
||||
};
|
||||
let wide = HuginnMindRequest::View { instance: slug(INSTANCE), id: cut_ref("wide") };
|
||||
let fits = HuginnMindRequest::View { instance: slug(INSTANCE), id: cut_ref("fits") };
|
||||
let (wide_answer, wide_size) = measured(&wide, &mut daemon);
|
||||
let (fits_answer, fits_size) = measured(&fits, &mut daemon);
|
||||
assert!(matches!(wide_answer, HuginnMindResponse::View(Some(_))), "{wide_answer:?}");
|
||||
assert!(wide_size > MAX_RESPONSE_BYTES, "the wide answer must be over one send, or nothing was deferred");
|
||||
assert!(fits_size <= MAX_RESPONSE_BYTES, "the narrow answer must fit one send");
|
||||
|
||||
let server = serve(root, daemon);
|
||||
let client = client(server.addr);
|
||||
assert_eq!(client.call(wide).unwrap(), wide_answer);
|
||||
assert_eq!(client.call(fits).unwrap(), fits_answer);
|
||||
}
|
||||
|
||||
/// Nothing listens on the port. The call fails naming the endpoint, and it
|
||||
/// fails when the configured timeout says: two timeouts, two waits.
|
||||
#[test]
|
||||
fn a_closed_port_is_unavailable_naming_the_endpoint_within_the_timeout() {
|
||||
let closed = UdpSocket::bind("127.0.0.1:0").unwrap().local_addr().unwrap();
|
||||
let mut waited = Vec::new();
|
||||
for timeout in [Duration::from_millis(300), Duration::from_millis(1200)] {
|
||||
let started = Instant::now();
|
||||
let error = HuginnClient::new(closed, slug(INSTANCE), timeout).call(HuginnMindRequest::Whoami).unwrap_err();
|
||||
let elapsed = started.elapsed();
|
||||
assert!(elapsed >= timeout, "gave up after {elapsed:?}, before its {timeout:?}");
|
||||
assert!(elapsed < timeout + Duration::from_millis(800), "gave up after {elapsed:?}, long past its {timeout:?}");
|
||||
assert!(error.to_string().contains(&closed.to_string()), "{error}");
|
||||
let (endpoint, detail) = unavailable(error);
|
||||
assert_eq!(endpoint, closed);
|
||||
assert!(detail.contains("no accept") && detail.contains("deadline"), "{detail}");
|
||||
waited.push(elapsed);
|
||||
}
|
||||
assert!(waited[1] > waited[0] + Duration::from_millis(500), "{waited:?}");
|
||||
}
|
||||
|
||||
/// A daemon that accepts and then says nothing is a timeout on the answer,
|
||||
/// not on the connection.
|
||||
#[test]
|
||||
fn a_daemon_that_never_answers_is_unavailable_within_the_timeout() {
|
||||
let server = scripted(|_| {
|
||||
std::thread::sleep(Duration::from_secs(3));
|
||||
CultNetMessage::Error { error: "late".into(), code: None, details: None }
|
||||
});
|
||||
let started = Instant::now();
|
||||
let error = HuginnClient::new(server.addr, slug(INSTANCE), Duration::from_millis(400))
|
||||
.call(HuginnMindRequest::Whoami)
|
||||
.unwrap_err();
|
||||
let elapsed = started.elapsed();
|
||||
assert!(elapsed >= Duration::from_millis(400) && elapsed < Duration::from_secs(2), "{elapsed:?}");
|
||||
let (endpoint, detail) = unavailable(error);
|
||||
assert_eq!(endpoint, server.addr);
|
||||
assert!(detail.contains("no answer"), "{detail}");
|
||||
}
|
||||
|
||||
/// An envelope failure means the request never reached a mind. It is the
|
||||
/// daemon refusing, not a transport failure, so a caller that retries on
|
||||
/// `Unavailable` does not loop on it; the code and the message arrive intact.
|
||||
#[test]
|
||||
fn an_envelope_failure_is_rejected_and_carries_its_code() {
|
||||
let server = answering(encode_failure(
|
||||
"eureka-state-call",
|
||||
"whoami",
|
||||
&OperationFailure { code: "wrong-service".into(), message: "not huginn.mind".into() },
|
||||
"scripted",
|
||||
));
|
||||
let error = client(server.addr).call(HuginnMindRequest::Whoami).unwrap_err();
|
||||
assert!(error.to_string().contains("wrong-service"), "{error}");
|
||||
let (endpoint, code, detail) = rejected(error);
|
||||
assert_eq!((endpoint, code.as_str(), detail.as_str()), (server.addr, "wrong-service", "not huginn.mind"));
|
||||
}
|
||||
|
||||
/// An answer to another request, or to another operation, is not this call's
|
||||
/// answer, whatever it says.
|
||||
#[test]
|
||||
fn an_answer_that_is_not_to_this_request_is_refused() {
|
||||
let foreign = answering(nothing_found("someone-elses-call", "whoami"));
|
||||
let (_, detail) = unavailable(client(foreign.addr).call(HuginnMindRequest::Whoami).unwrap_err());
|
||||
assert!(detail.contains("someone-elses-call"), "{detail}");
|
||||
|
||||
let other_operation = answering(nothing_found("eureka-state-call", "query"));
|
||||
let (_, detail) = unavailable(client(other_operation.addr).call(HuginnMindRequest::Whoami).unwrap_err());
|
||||
assert!(detail.contains("query") && detail.contains("whoami"), "{detail}");
|
||||
|
||||
let matching = answering(nothing_found("eureka-state-call", "whoami"));
|
||||
assert_eq!(client(matching.addr).call(HuginnMindRequest::Whoami).unwrap(), HuginnMindResponse::View(None));
|
||||
}
|
||||
|
||||
/// The timeout bounds the whole call. Seven chunks each answered after 300 ms
|
||||
/// take two seconds; no single wait exceeds the 500 ms timeout, and the call
|
||||
/// still gives up at it.
|
||||
#[test]
|
||||
fn the_timeout_bounds_the_whole_call_not_each_wait() {
|
||||
let server = deferring_in_chunks(vec![7_u8; 7 * 1024], 1024, Duration::from_millis(300));
|
||||
let timeout = Duration::from_millis(500);
|
||||
let started = Instant::now();
|
||||
let error = HuginnClient::new(server.addr, slug(INSTANCE), timeout).call(HuginnMindRequest::Whoami).unwrap_err();
|
||||
let elapsed = started.elapsed();
|
||||
assert!(elapsed >= timeout && elapsed < Duration::from_millis(1100), "{elapsed:?}");
|
||||
let (_, detail) = unavailable(error);
|
||||
assert!(detail.contains("deadline"), "{detail}");
|
||||
}
|
||||
|
||||
/// Every call ends its session, whether it answered, was rejected, or gave up.
|
||||
#[test]
|
||||
fn every_call_disconnects_its_session() {
|
||||
let answered = answering(nothing_found("eureka-state-call", "whoami"));
|
||||
client(answered.addr).call(HuginnMindRequest::Whoami).unwrap();
|
||||
assert!(ended_within(&answered, 1, Duration::from_secs(5)), "an answered call left its session open");
|
||||
|
||||
let refused = answering(encode_failure(
|
||||
"eureka-state-call",
|
||||
"whoami",
|
||||
&OperationFailure { code: "wrong-service".into(), message: "no".into() },
|
||||
"scripted",
|
||||
));
|
||||
rejected(client(refused.addr).call(HuginnMindRequest::Whoami).unwrap_err());
|
||||
assert!(ended_within(&refused, 1, Duration::from_secs(5)), "a rejected call left its session open");
|
||||
|
||||
let silent = scripted(|_| {
|
||||
std::thread::sleep(Duration::from_millis(800));
|
||||
CultNetMessage::Error { error: "late".into(), code: None, details: None }
|
||||
});
|
||||
let error = HuginnClient::new(silent.addr, slug(INSTANCE), Duration::from_millis(300))
|
||||
.call(HuginnMindRequest::Whoami)
|
||||
.unwrap_err();
|
||||
unavailable(error);
|
||||
assert!(ended_within(&silent, 1, Duration::from_secs(5)), "a timed-out call left its session open");
|
||||
}
|
||||
|
||||
/// A deferred body that is not a response, and one that is another deferral,
|
||||
/// are protocol violations named as such.
|
||||
#[test]
|
||||
fn a_deferred_body_must_be_a_response_and_not_another_deferral() {
|
||||
let garbage = deferring(vec![0xc1; 40_000]);
|
||||
let (_, detail) = unavailable(client(garbage.addr).call(HuginnMindRequest::Whoami).unwrap_err());
|
||||
assert!(detail.contains("not a response"), "{detail}");
|
||||
|
||||
let (inner, _) = pack_content("inner", "package", "", "", "", b"x", 16).unwrap();
|
||||
let again = rmp_serde::to_vec_named(&HuginnMindResponse::Deferred(DeferredAnswer { manifest: inner })).unwrap();
|
||||
let nested = deferring(again);
|
||||
let (_, detail) = unavailable(client(nested.addr).call(HuginnMindRequest::Whoami).unwrap_err());
|
||||
assert!(detail.contains("itself deferred"), "{detail}");
|
||||
}
|
||||
|
||||
/// A body over the deferral cap is refused before its first chunk is asked
|
||||
/// for, by the number the daemon defers under.
|
||||
#[test]
|
||||
fn a_body_over_the_deferral_cap_is_refused() {
|
||||
let over = huginn_mind::MAX_DEFERRED_BODY_BYTES as usize + 1;
|
||||
let server = deferring(vec![0_u8; over]);
|
||||
let (_, detail) = unavailable(client(server.addr).call(HuginnMindRequest::Whoami).unwrap_err());
|
||||
assert!(detail.contains("exceeds"), "{detail}");
|
||||
}
|
||||
|
||||
/// An admission whose session is `session_pad` characters long: the knob that
|
||||
/// sizes a request without changing what it asks.
|
||||
fn padded_admit(session_pad: usize) -> HuginnMindRequest {
|
||||
let mut request = batch(INSTANCE, vec![stewardship()]);
|
||||
request.provenance.session = Short("s".repeat(session_pad));
|
||||
HuginnMindRequest::Admit(request)
|
||||
}
|
||||
|
||||
/// What one send has to carry for `message`, measured the way the client does.
|
||||
fn encoded_bytes(message: &CultNetMessage) -> usize {
|
||||
encode_cultnet_message_to_vec(message, CultNetWireContract::CultNetSchemaV0).unwrap().len()
|
||||
}
|
||||
|
||||
fn message_for(request: &HuginnMindRequest, source: Option<String>) -> CultNetMessage {
|
||||
encode_request("eureka-state-call", request, source).unwrap()
|
||||
}
|
||||
|
||||
/// The smallest session padding whose request encodes to at least `target`
|
||||
/// bytes. The payload rides as base64, so request sizes step by four and only
|
||||
/// every fourth size is reachable through the request alone.
|
||||
fn pad_for(target: usize) -> usize {
|
||||
let (mut low, mut high) = (0, target);
|
||||
while low < high {
|
||||
let middle = (low + high) / 2;
|
||||
if encoded_bytes(&message_for(&padded_admit(middle), None)) < target {
|
||||
low = middle + 1;
|
||||
} else {
|
||||
high = middle;
|
||||
}
|
||||
}
|
||||
low
|
||||
}
|
||||
|
||||
/// The largest request one send carries, and the smallest it does not.
|
||||
fn largest_request_that_fits() -> HuginnMindRequest {
|
||||
padded_admit(pad_for(MAX_REQUEST_BYTES + 1) - 1)
|
||||
}
|
||||
|
||||
fn smallest_request_over() -> HuginnMindRequest {
|
||||
padded_admit(pad_for(MAX_REQUEST_BYTES + 1))
|
||||
}
|
||||
|
||||
/// A message that encodes to exactly `target` bytes: a request a little under
|
||||
/// it, topped up with a source runtime id, which rides outside the payload and
|
||||
/// moves the size a byte at a time once the field is there at all.
|
||||
fn message_of_size(target: usize) -> CultNetMessage {
|
||||
let fitting = pad_for(target + 1) - 1;
|
||||
for padding in (fitting.saturating_sub(80)..=fitting).rev() {
|
||||
let request = padded_admit(padding);
|
||||
for source in 1..80 {
|
||||
let message = message_for(&request, Some("r".repeat(source)));
|
||||
if encoded_bytes(&message) == target {
|
||||
return message;
|
||||
}
|
||||
}
|
||||
}
|
||||
panic!("no message encodes to {target} bytes");
|
||||
}
|
||||
|
||||
fn admitted_answer() -> CultNetMessage {
|
||||
encode_response(
|
||||
"eureka-state-call",
|
||||
"admit",
|
||||
&HuginnMindResponse::Admit(PipelineAdmissionOutcome::AlreadyAdmitted { receipt_id: "r".into() }),
|
||||
"scripted",
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// A request larger than one send carries is refused by the client before it
|
||||
/// connects to anything, as its own error and not as a transport failure the
|
||||
/// caller would retry; the largest request that fits is sent, and answered.
|
||||
#[test]
|
||||
fn a_request_over_what_one_send_carries_is_too_large_and_one_that_fits_is_sent() {
|
||||
let closed = UdpSocket::bind("127.0.0.1:0").unwrap().local_addr().unwrap();
|
||||
let over = smallest_request_over();
|
||||
let started = Instant::now();
|
||||
match client(closed).call(over.clone()) {
|
||||
Err(ClientError::TooLarge { bytes, limit }) => {
|
||||
assert_eq!(bytes, encoded_bytes(&message_for(&over, None)));
|
||||
assert_eq!(limit, MAX_REQUEST_BYTES);
|
||||
assert!(bytes > limit);
|
||||
}
|
||||
other => panic!("expected TooLarge, got {other:?}"),
|
||||
}
|
||||
assert!(started.elapsed() < Duration::from_secs(1), "refused only after trying the transport");
|
||||
|
||||
let fits = largest_request_that_fits();
|
||||
assert!(MAX_REQUEST_BYTES - encoded_bytes(&message_for(&fits, None)) < 4, "not the largest that fits");
|
||||
let server = answering(admitted_answer());
|
||||
let answer = client(server.addr).call(fits);
|
||||
assert!(matches!(answer, Ok(HuginnMindResponse::Admit(PipelineAdmissionOutcome::AlreadyAdmitted { .. }))), "{answer:?}");
|
||||
}
|
||||
|
||||
/// The transport's own refusal of an oversize send ("Message too long") is
|
||||
/// what an unchecked request used to report, as an `Unavailable` daemon a
|
||||
/// caller would retry forever.
|
||||
#[test]
|
||||
fn an_oversize_admission_is_never_reported_as_an_unavailable_daemon() {
|
||||
let server = answering(admitted_answer());
|
||||
let error = client(server.addr).call(padded_admit(70_000)).unwrap_err();
|
||||
assert!(matches!(error, ClientError::TooLarge { .. }), "{error}");
|
||||
}
|
||||
|
||||
/// The limit is the transport's own: a message of `MAX_REQUEST_BYTES` goes out
|
||||
/// on an RUDP session, and one byte more is refused by the socket.
|
||||
#[test]
|
||||
fn the_limit_is_the_largest_message_the_transport_sends() {
|
||||
let server = answering(admitted_answer());
|
||||
let mut session = CultMesh::create_rudp_client_for_endpoint(
|
||||
"limit-probe".to_string(),
|
||||
CULTNET_OPERATION_CONNECTION_ID,
|
||||
&format!("rudp://{}", server.addr),
|
||||
CultMeshRudpSocketOptions::default(),
|
||||
)
|
||||
.unwrap();
|
||||
session.connect(Vec::new()).unwrap();
|
||||
let until = Instant::now() + Duration::from_secs(5);
|
||||
while !session.connected() {
|
||||
session.poll_resends().unwrap();
|
||||
let _ = session.receive_once().unwrap();
|
||||
assert!(Instant::now() < until, "the server never accepted");
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
session.send_schema_message(&message_of_size(MAX_REQUEST_BYTES)).expect("the limit is carried");
|
||||
assert!(session.send_schema_message(&message_of_size(MAX_REQUEST_BYTES + 1)).is_err(), "one byte more is sent");
|
||||
}
|
||||
185
crates/eureka-state/tests/common/mod.rs
Normal file
185
crates/eureka-state/tests/common/mod.rs
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
//! The daemon half of the fixtures: an in-process Huginn daemon over a
|
||||
//! temporary mind, on a loopback port, shared by the client and the server
|
||||
//! tests.
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
|
||||
use std::thread::JoinHandle;
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::Utc;
|
||||
use cultnet_rs::{CultNetMessage, CultNetRudpServerEvent, CultNetWireContract, decode_cultnet_message_from_slice};
|
||||
use huginn_daemon::serve::{bind, run, schema_registry};
|
||||
use huginn_daemon::{Daemon, Handled, Hits, IndexSink, SearchTicket, ServeOptions};
|
||||
use huginn_mind::epiphany_pipeline::{
|
||||
Date, DocRef, OrgRepo, PipelineCampaign, PipelineDocument, PipelineInstance, PipelineRef, PipelineStewardship, Sha, Short, Slug,
|
||||
Title,
|
||||
};
|
||||
use huginn_mind::{
|
||||
Faculty, HuginnMindRequest, HuginnMindResponse, IndexStatus, Mind, MindStore, OwnedRedbMessagePackBackingStore,
|
||||
PipelineAdmissionBatch, PipelineAdmissionOutcome, PipelineProvenance,
|
||||
};
|
||||
use tempfile::TempDir;
|
||||
|
||||
pub const INSTANCE: &str = "eureka";
|
||||
|
||||
pub fn slug(value: &str) -> Slug {
|
||||
Slug(value.into())
|
||||
}
|
||||
|
||||
pub fn identity(name: &str) -> PipelineDocument {
|
||||
PipelineDocument::Instance(PipelineInstance {
|
||||
instance: slug(name),
|
||||
display_name: Short(format!("{name} mind")),
|
||||
created_at: Date("2026-09-29".into()),
|
||||
host: Short(name.into()),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn batch(instance: &str, documents: Vec<PipelineDocument>) -> PipelineAdmissionBatch {
|
||||
PipelineAdmissionBatch {
|
||||
instance: slug(instance),
|
||||
provenance: PipelineProvenance {
|
||||
faculty: Faculty::Hands,
|
||||
agent: Short("claude".into()),
|
||||
session: Short("session-1".into()),
|
||||
tool: Short("admit".into()),
|
||||
},
|
||||
documents,
|
||||
}
|
||||
}
|
||||
|
||||
/// No index: the client's subject is the transport, not the projection.
|
||||
pub struct Inert;
|
||||
|
||||
impl<S: MindStore> IndexSink<S> for Inert {
|
||||
fn committed(&mut self, _mind: &Mind<S>, _writes: &[PipelineRef]) -> anyhow::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
fn status(&self) -> IndexStatus {
|
||||
IndexStatus::Current
|
||||
}
|
||||
fn search(&mut self, _text: &str, _top_k: u32) -> Result<SearchTicket, String> {
|
||||
Err("this index cannot search".into())
|
||||
}
|
||||
fn search_deadline(&self) -> Duration {
|
||||
Duration::ZERO
|
||||
}
|
||||
fn searched(&mut self) -> Vec<(SearchTicket, Result<Hits, String>)> {
|
||||
Vec::new()
|
||||
}
|
||||
fn abandon(&mut self, _ticket: SearchTicket) {}
|
||||
}
|
||||
|
||||
pub type TestDaemon = Daemon<OwnedRedbMessagePackBackingStore, Inert>;
|
||||
|
||||
pub fn answered(handled: Handled) -> HuginnMindResponse {
|
||||
match handled {
|
||||
Handled::Answered(response) => response,
|
||||
Handled::Searching(_) => panic!("the inert index cannot search"),
|
||||
}
|
||||
}
|
||||
|
||||
/// A daemon over a temporary mind whose identity is admitted, and whatever
|
||||
/// else `seed` admits.
|
||||
pub fn mind(seed: Vec<Vec<PipelineDocument>>) -> (TempDir, TestDaemon) {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut daemon = Daemon::new(Mind::open(root.path(), &slug(INSTANCE)).unwrap(), Inert);
|
||||
for documents in std::iter::once(vec![identity(INSTANCE)]).chain(seed) {
|
||||
let outcome = answered(daemon.handle(HuginnMindRequest::Admit(batch(INSTANCE, documents)), Utc::now()));
|
||||
assert!(matches!(outcome, HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { .. })), "{outcome:?}");
|
||||
}
|
||||
(root, daemon)
|
||||
}
|
||||
|
||||
/// A server on a loopback port that stops when dropped.
|
||||
pub struct Server {
|
||||
pub addr: SocketAddr,
|
||||
pub stopping: Arc<AtomicBool>,
|
||||
pub thread: Option<JoinHandle<()>>,
|
||||
/// The sessions a scripted server has seen end: what the client's
|
||||
/// disconnect looks like from the daemon's side.
|
||||
pub ended: Arc<AtomicUsize>,
|
||||
pub _root: Option<TempDir>,
|
||||
}
|
||||
|
||||
impl Drop for Server {
|
||||
fn drop(&mut self) {
|
||||
self.stopping.store(true, Ordering::Relaxed);
|
||||
if let Some(thread) = self.thread.take() {
|
||||
let _ = thread.join();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn serve(root: TempDir, mut daemon: TestDaemon) -> Server {
|
||||
let mut hub = bind("127.0.0.1:0".parse().unwrap(), &daemon.runtime_id()).unwrap();
|
||||
let addr = hub.local_addr().unwrap();
|
||||
let stopping = Arc::new(AtomicBool::new(false));
|
||||
let loop_stopping = Arc::clone(&stopping);
|
||||
let registry = schema_registry().unwrap();
|
||||
let thread = std::thread::spawn(move || {
|
||||
run(&mut daemon, &mut hub, ®istry, &loop_stopping, &ServeOptions::default()).unwrap();
|
||||
});
|
||||
Server { addr, stopping, thread: Some(thread), ended: Arc::default(), _root: Some(root) }
|
||||
}
|
||||
|
||||
/// A server that answers each schema message with whatever `script` says.
|
||||
pub fn scripted(mut script: impl FnMut(&CultNetMessage) -> CultNetMessage + Send + 'static) -> Server {
|
||||
let mut hub = bind("127.0.0.1:0".parse().unwrap(), "scripted").unwrap();
|
||||
let addr = hub.local_addr().unwrap();
|
||||
let stopping = Arc::new(AtomicBool::new(false));
|
||||
let loop_stopping = Arc::clone(&stopping);
|
||||
let ended = Arc::new(AtomicUsize::new(0));
|
||||
let loop_ended = Arc::clone(&ended);
|
||||
let thread = std::thread::spawn(move || {
|
||||
while !loop_stopping.load(Ordering::Relaxed) {
|
||||
hub.poll_resends().unwrap();
|
||||
while let Some(event) = hub.receive_event_once().unwrap() {
|
||||
if matches!(event, CultNetRudpServerEvent::Disconnected { .. }) {
|
||||
loop_ended.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
let CultNetRudpServerEvent::Frame { session, frame } = event else { continue };
|
||||
let message =
|
||||
decode_cultnet_message_from_slice(&frame.payload, CultNetWireContract::CultNetSchemaV0).unwrap();
|
||||
hub.send_schema_message(&session, &script(&message)).unwrap();
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
});
|
||||
Server { addr, stopping, thread: Some(thread), ended, _root: None }
|
||||
}
|
||||
|
||||
/// A scripted server whose every answer is `reply`.
|
||||
pub fn answering(reply: CultNetMessage) -> Server {
|
||||
scripted(move |_| reply.clone())
|
||||
}
|
||||
|
||||
pub const CAMPAIGN: &str = "eureka-state";
|
||||
pub const REPO: &str = "GameCult/Epiphany";
|
||||
|
||||
pub fn sha() -> Sha {
|
||||
Sha("5f98228d".into())
|
||||
}
|
||||
|
||||
pub fn campaign() -> PipelineDocument {
|
||||
PipelineDocument::Campaign(PipelineCampaign {
|
||||
slug: slug(CAMPAIGN),
|
||||
title: Title("Eureka pipeline state".into()),
|
||||
repos: vec![OrgRepo(REPO.into())],
|
||||
working_branch: Short("hands/cut13a".into()),
|
||||
target_doc: DocRef { path: Short("notes/target.md".into()), start_line: 1, end_line: 9, commit: sha() },
|
||||
})
|
||||
}
|
||||
|
||||
pub fn stewardship() -> PipelineDocument {
|
||||
PipelineDocument::Stewardship(PipelineStewardship {
|
||||
instance: slug(INSTANCE),
|
||||
repo: OrgRepo(REPO.into()),
|
||||
sequence: 1,
|
||||
assigned_on: Date("2026-09-29".into()),
|
||||
note: "assigned".into(),
|
||||
})
|
||||
}
|
||||
532
crates/eureka-state/tests/mcp.rs
Normal file
532
crates/eureka-state/tests/mcp.rs
Normal file
|
|
@ -0,0 +1,532 @@
|
|||
//! The `eureka-state` binary over stdio, as Claude Code drives it: real
|
||||
//! processes, real JSON-RPC, a real daemon in-process behind it (or a scripted
|
||||
//! server where the daemon cannot be made to misbehave).
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::io::{BufRead, BufReader, Write};
|
||||
use std::net::UdpSocket;
|
||||
use std::process::{Child, ChildStdin, Command, Stdio};
|
||||
use std::sync::mpsc::{Receiver, channel};
|
||||
use std::time::Duration;
|
||||
|
||||
use huginn_mind::envelope::{OperationFailure, encode_failure, encode_response};
|
||||
use huginn_mind::{HuginnMindRequest, HuginnMindResponse};
|
||||
use huginn_mind::epiphany_pipeline::{PipelineKind, PipelineRef, Short};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
mod common;
|
||||
use common::*;
|
||||
|
||||
const WAIT: Duration = Duration::from_secs(60);
|
||||
|
||||
struct Mcp {
|
||||
child: Child,
|
||||
stdin: ChildStdin,
|
||||
lines: Receiver<String>,
|
||||
next: u64,
|
||||
/// The server's answer to `initialize`.
|
||||
init: Value,
|
||||
}
|
||||
|
||||
impl Drop for Mcp {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.child.kill();
|
||||
let _ = self.child.wait();
|
||||
}
|
||||
}
|
||||
|
||||
impl Mcp {
|
||||
/// The binary with exactly the environment given, initialised.
|
||||
fn start(env: &[(&str, &str)]) -> Self {
|
||||
Self::spawn(|command| {
|
||||
command.envs(env.iter().copied());
|
||||
})
|
||||
}
|
||||
|
||||
/// The binary with no configuration but what `configure` gives its command.
|
||||
fn spawn(configure: impl FnOnce(&mut Command)) -> Self {
|
||||
let mut command = Command::new(env!("CARGO_BIN_EXE_eureka-state"));
|
||||
command.env_remove("EUREKA_INSTANCE").env_remove("HUGINN_ENDPOINT");
|
||||
configure(&mut command);
|
||||
command
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::null());
|
||||
let mut child = command.spawn().expect("the binary starts");
|
||||
let stdin = child.stdin.take().unwrap();
|
||||
let stdout = child.stdout.take().unwrap();
|
||||
let (sender, lines) = channel();
|
||||
std::thread::spawn(move || {
|
||||
for line in BufReader::new(stdout).lines().map_while(Result::ok) {
|
||||
let _ = sender.send(line);
|
||||
}
|
||||
});
|
||||
let mut mcp = Self { child, stdin, lines, next: 1, init: Value::Null };
|
||||
let init = mcp.rpc(
|
||||
"initialize",
|
||||
json!({ "protocolVersion": "2025-06-18", "capabilities": {}, "clientInfo": { "name": "test", "version": "0" } }),
|
||||
);
|
||||
assert!(init.get("result").is_some(), "{init}");
|
||||
mcp.init = init["result"].clone();
|
||||
mcp.send(json!({ "jsonrpc": "2.0", "method": "notifications/initialized" }));
|
||||
mcp
|
||||
}
|
||||
|
||||
/// Configured for `instance` against a daemon at `addr`.
|
||||
fn at(instance: &str, addr: std::net::SocketAddr) -> Self {
|
||||
Self::start(&[("EUREKA_INSTANCE", instance), ("HUGINN_ENDPOINT", &format!("rudp://{addr}"))])
|
||||
}
|
||||
|
||||
fn send(&mut self, message: Value) {
|
||||
writeln!(self.stdin, "{message}").unwrap();
|
||||
self.stdin.flush().unwrap();
|
||||
}
|
||||
|
||||
/// One request and its response, whole (`result` or `error`).
|
||||
fn rpc(&mut self, method: &str, params: Value) -> Value {
|
||||
let id = self.next;
|
||||
self.next += 1;
|
||||
self.send(json!({ "jsonrpc": "2.0", "id": id, "method": method, "params": params }));
|
||||
loop {
|
||||
let line = self.lines.recv_timeout(WAIT).expect("the server answers");
|
||||
let message: Value = serde_json::from_str(&line).expect("stdout carries only JSON-RPC");
|
||||
if message["id"] == json!(id) {
|
||||
return message;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A tool call's result: whether it is an error, and its JSON body.
|
||||
fn call(&mut self, tool: &str, arguments: Value) -> (bool, Value) {
|
||||
let response = self.rpc("tools/call", json!({ "name": tool, "arguments": arguments }));
|
||||
let result = &response["result"];
|
||||
assert!(result.is_object(), "a tool result, not a protocol error: {response}");
|
||||
let text = result["content"][0]["text"].as_str().expect("text content");
|
||||
(result["isError"] == json!(true), serde_json::from_str(text).expect("a JSON body"))
|
||||
}
|
||||
}
|
||||
|
||||
/// A loopback UDP port nothing listens on.
|
||||
fn closed_port() -> std::net::SocketAddr {
|
||||
UdpSocket::bind("127.0.0.1:0").unwrap().local_addr().unwrap()
|
||||
}
|
||||
|
||||
fn faculty_args(documents: Vec<Value>) -> Value {
|
||||
json!({ "faculty": "Hands", "agent": "claude", "session": "session-7", "documents": documents })
|
||||
}
|
||||
|
||||
fn campaign_json() -> Value {
|
||||
serde_json::to_value(campaign()).unwrap()
|
||||
}
|
||||
|
||||
/// Every `$ref` in a schema, and the `$defs` it can resolve against.
|
||||
fn refs(schema: &Value, found: &mut Vec<String>) {
|
||||
match schema {
|
||||
Value::Object(map) => {
|
||||
for (key, value) in map {
|
||||
if key == "$ref" {
|
||||
found.push(value.as_str().unwrap().to_string());
|
||||
} else {
|
||||
refs(value, found);
|
||||
}
|
||||
}
|
||||
}
|
||||
Value::Array(items) => items.iter().for_each(|item| refs(item, found)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
/// Every key named `key` anywhere below `schema`'s root.
|
||||
fn keys_below_root(schema: &Value, key: &str) -> usize {
|
||||
fn count(value: &Value, key: &str) -> usize {
|
||||
match value {
|
||||
Value::Object(map) => map.iter().map(|(name, inner)| usize::from(name == key) + count(inner, key)).sum(),
|
||||
Value::Array(items) => items.iter().map(|item| count(item, key)).sum(),
|
||||
_ => 0,
|
||||
}
|
||||
}
|
||||
schema.as_object().unwrap().values().map(|inner| count(inner, key)).sum()
|
||||
}
|
||||
|
||||
/// Four tools, each input schema a self-contained document: every `$ref` is
|
||||
/// internal and resolves, and the selection's definitions are bundled in.
|
||||
#[test]
|
||||
fn tools_list_offers_four_tools_with_self_contained_schemas() {
|
||||
let mut mcp = Mcp::at(INSTANCE, closed_port());
|
||||
let listed = mcp.rpc("tools/list", json!({}));
|
||||
let tools = listed["result"]["tools"].as_array().expect("tools");
|
||||
let names: BTreeSet<&str> = tools.iter().map(|tool| tool["name"].as_str().unwrap()).collect();
|
||||
assert_eq!(names, BTreeSet::from(["whoami", "admit", "view", "query"]));
|
||||
for tool in tools {
|
||||
let schema = &tool["inputSchema"];
|
||||
let mut found = Vec::new();
|
||||
refs(schema, &mut found);
|
||||
for reference in &found {
|
||||
let name = reference.strip_prefix("#/$defs/").unwrap_or_else(|| panic!("{}: remote $ref {reference}", tool["name"]));
|
||||
assert!(schema["$defs"].get(name).is_some(), "{}: {reference} does not resolve", tool["name"]);
|
||||
}
|
||||
// The vendored selection schema brings its own `$id` and `$schema`,
|
||||
// which would make its `$ref`s resolve against a remote document.
|
||||
assert!(schema.get("$id").is_none(), "{}: a remote identity", tool["name"]);
|
||||
for key in ["$id", "$schema"] {
|
||||
assert_eq!(keys_below_root(schema, key), 0, "{}: a nested {key}", tool["name"]);
|
||||
}
|
||||
}
|
||||
let query = tools.iter().find(|tool| tool["name"] == "query").unwrap();
|
||||
for bundled in ["fieldPredicate", "citation", "incoming", "recordRef"] {
|
||||
assert!(query["inputSchema"]["$defs"].get(bundled).is_some(), "{bundled} is not bundled");
|
||||
}
|
||||
assert_eq!(query["inputSchema"]["properties"]["selection"]["properties"]["limit"]["maximum"], json!(200));
|
||||
}
|
||||
|
||||
/// The vendored selection schema is CultLib's, at the revision `cultnet-rs`
|
||||
/// is built from.
|
||||
#[test]
|
||||
fn selection_schema_matches_cultlib() {
|
||||
let cargo = std::env::var("CARGO").unwrap_or_else(|_| "cargo".into());
|
||||
let metadata = Command::new(cargo)
|
||||
.args(["metadata", "--format-version", "1", "--offline", "--locked"])
|
||||
.current_dir(env!("CARGO_MANIFEST_DIR"))
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(metadata.status.success(), "{}", String::from_utf8_lossy(&metadata.stderr));
|
||||
let metadata: Value = serde_json::from_slice(&metadata.stdout).unwrap();
|
||||
let manifest = metadata["packages"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.find(|package| package["name"] == "cultnet-rs")
|
||||
.expect("cultnet-rs is a dependency")["manifest_path"]
|
||||
.as_str()
|
||||
.unwrap();
|
||||
let crate_dir = std::path::Path::new(manifest).parent().unwrap();
|
||||
let published = crate_dir.join("../../contracts/cultnet/cultnet.selection.schema.json");
|
||||
let theirs: Value = serde_json::from_str(&std::fs::read_to_string(published).unwrap()).unwrap();
|
||||
let ours: Value = serde_json::from_str(
|
||||
&std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/schemas/cultnet.selection.schema.json")).unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(ours, theirs);
|
||||
}
|
||||
|
||||
/// Every tool against a live daemon, and what each returns.
|
||||
#[test]
|
||||
fn every_tool_round_trips_against_a_live_daemon() {
|
||||
let (root, daemon) = mind(vec![vec![stewardship()]]);
|
||||
let server = serve(root, daemon);
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
|
||||
let (error, whoami) = mcp.call("whoami", json!({}));
|
||||
assert!(!error);
|
||||
assert_eq!(whoami["instance"], json!(INSTANCE));
|
||||
assert_eq!(whoami["endpoint"], json!(format!("rudp://{}", server.addr)));
|
||||
assert_eq!(whoami["reachable"], json!(true));
|
||||
assert_eq!(whoami["status"]["instance"], json!(INSTANCE));
|
||||
assert_eq!(whoami["status"]["documents"], json!(2));
|
||||
assert!(whoami.get("error").is_none(), "{whoami}");
|
||||
|
||||
let (error, admitted) = mcp.call("admit", faculty_args(vec![campaign_json()]));
|
||||
assert!(!error);
|
||||
let writes = admitted["Committed"]["writes"].as_array().expect("committed").clone();
|
||||
assert_eq!(writes.len(), 1, "{admitted}");
|
||||
|
||||
let (error, viewed) = mcp.call("view", json!({ "id": writes[0] }));
|
||||
assert!(!error);
|
||||
assert_eq!(viewed["id"], writes[0]);
|
||||
assert_eq!(viewed["admission"]["provenance"]["session"], json!("session-7"));
|
||||
|
||||
let (error, nothing) = mcp.call("view", json!({ "id": absent() }));
|
||||
assert!(!error);
|
||||
assert_eq!(nothing, Value::Null);
|
||||
|
||||
let (error, page) = mcp.call("query", json!({ "selection": { "projection": "document" } }));
|
||||
assert!(!error);
|
||||
assert_eq!(page["matched"], json!(3), "{page}");
|
||||
assert_eq!(page["items"]["Documents"].as_array().unwrap().len(), 3);
|
||||
|
||||
let (_, counted) = mcp.call("whoami", json!({}));
|
||||
assert_eq!(counted["status"]["receipts"], json!(3));
|
||||
}
|
||||
|
||||
/// Declared identity is configuration, not a per-call argument. Input that
|
||||
/// names another instance, or another tool, changes neither: the batch is
|
||||
/// admitted for the configured instance, and the receipt says `eureka-state`.
|
||||
#[test]
|
||||
fn admit_ignores_any_instance_in_input() {
|
||||
let (root, daemon) = mind(vec![vec![stewardship()]]);
|
||||
let server = serve(root, daemon);
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
|
||||
let mut arguments = faculty_args(vec![campaign_json()]);
|
||||
arguments["instance"] = json!("thought-cage");
|
||||
arguments["tool"] = json!("someone-else");
|
||||
arguments["provenance"] = json!({ "faculty": "Operator", "agent": "x", "session": "y", "tool": "someone-else" });
|
||||
let (error, admitted) = mcp.call("admit", arguments);
|
||||
assert!(!error);
|
||||
let writes = admitted["Committed"]["writes"].as_array().unwrap_or_else(|| panic!("not committed: {admitted}"));
|
||||
|
||||
let (_, viewed) = mcp.call("view", json!({ "id": writes[0] }));
|
||||
let provenance = &viewed["admission"]["provenance"];
|
||||
assert_eq!(provenance["tool"], json!("eureka-state"));
|
||||
assert_eq!(provenance["faculty"], json!("Hands"));
|
||||
assert_eq!(provenance["agent"], json!("claude"));
|
||||
assert_eq!(provenance["session"], json!("session-7"));
|
||||
}
|
||||
|
||||
/// A mind's refusal is an answer: a successful result carrying the typed
|
||||
/// refusal. Here the configured instance is not the daemon's.
|
||||
#[test]
|
||||
fn a_refusal_is_a_successful_result() {
|
||||
let (root, daemon) = mind(vec![vec![stewardship()]]);
|
||||
let server = serve(root, daemon);
|
||||
let mut mcp = Mcp::at("thought-cage", server.addr);
|
||||
|
||||
let (error, viewed) = mcp.call("view", json!({ "id": absent() }));
|
||||
assert!(!error);
|
||||
assert!(viewed.get("Refused").is_some(), "{viewed}");
|
||||
let (error, queried) = mcp.call("query", json!({ "selection": {} }));
|
||||
assert!(!error);
|
||||
assert!(queried.get("Refused").is_some(), "{queried}");
|
||||
let (error, admitted) = mcp.call("admit", faculty_args(vec![campaign_json()]));
|
||||
assert!(!error);
|
||||
assert!(admitted.get("Refused").is_some(), "{admitted}");
|
||||
}
|
||||
|
||||
/// A daemon that cannot be reached is `isError` with the endpoint, and
|
||||
/// `whoami` reports it as `reachable: false` without a protocol error.
|
||||
#[test]
|
||||
fn a_closed_port_is_reported_and_is_not_a_protocol_error() {
|
||||
let addr = closed_port();
|
||||
let mut mcp = Mcp::at(INSTANCE, addr);
|
||||
|
||||
let whoami = mcp.rpc("tools/call", json!({ "name": "whoami", "arguments": {} }));
|
||||
assert!(whoami.get("error").is_none(), "{whoami}");
|
||||
assert_ne!(whoami["result"]["isError"], json!(true));
|
||||
let body: Value = serde_json::from_str(whoami["result"]["content"][0]["text"].as_str().unwrap()).unwrap();
|
||||
assert_eq!(body["reachable"], json!(false));
|
||||
assert_eq!(body["status"], Value::Null);
|
||||
assert_eq!(body["error"]["error"], json!("Unavailable"));
|
||||
|
||||
let (error, view) = mcp.call("view", json!({ "id": absent() }));
|
||||
assert!(error);
|
||||
assert_eq!(view["error"], json!("Unavailable"));
|
||||
assert_eq!(view["endpoint"], json!(format!("rudp://{addr}")));
|
||||
}
|
||||
|
||||
/// A daemon rejecting the envelope is a permanent mismatch, not a down
|
||||
/// daemon: `isError` with its own code, distinct from `Unavailable`.
|
||||
#[test]
|
||||
fn a_rejected_request_is_a_typed_error_distinct_from_unavailable() {
|
||||
let server = answering(encode_failure(
|
||||
"eureka-state-call",
|
||||
"view",
|
||||
&OperationFailure { code: "wrong-service".into(), message: "not huginn.mind".into() },
|
||||
"scripted",
|
||||
));
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
let (error, body) = mcp.call("view", json!({ "id": absent() }));
|
||||
assert!(error);
|
||||
assert_eq!(body["error"], json!("Rejected"));
|
||||
assert_eq!(body["code"], json!("wrong-service"));
|
||||
assert_eq!(body["detail"], json!("not huginn.mind"));
|
||||
assert_eq!(body["endpoint"], json!(format!("rudp://{}", server.addr)));
|
||||
}
|
||||
|
||||
/// A `whoami` answered with another operation's payload is not a mind's
|
||||
/// status: it is reported, not called reachable.
|
||||
#[test]
|
||||
fn whoami_does_not_take_another_answer_for_a_status() {
|
||||
let server = answering(encode_response("eureka-state-call", "whoami", &HuginnMindResponse::View(None), "scripted").unwrap());
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
let (error, body) = mcp.call("whoami", json!({}));
|
||||
assert!(!error);
|
||||
assert_eq!(body["reachable"], json!(false));
|
||||
assert_eq!(body["status"], Value::Null);
|
||||
assert!(body["error"]["detail"].as_str().unwrap().contains("unexpected answer"), "{body}");
|
||||
}
|
||||
|
||||
/// Bad configuration does not stop the server. `whoami` says what is wrong and
|
||||
/// every other tool refuses with the same typed body.
|
||||
#[test]
|
||||
fn bad_configuration_is_served_and_reported() {
|
||||
let cases: [(&[(&str, &str)], &str); 5] = [
|
||||
(&[("HUGINN_ENDPOINT", "rudp://127.0.0.1:1")], "EUREKA_INSTANCE is not set"),
|
||||
(&[("EUREKA_INSTANCE", "Not A Slug"), ("HUGINN_ENDPOINT", "rudp://127.0.0.1:1")], "EUREKA_INSTANCE is not a valid"),
|
||||
(&[("EUREKA_INSTANCE", "eureka")], "HUGINN_ENDPOINT is not set"),
|
||||
(&[("EUREKA_INSTANCE", "eureka"), ("HUGINN_ENDPOINT", "127.0.0.1:1")], "HUGINN_ENDPOINT is not rudp://"),
|
||||
(&[("EUREKA_INSTANCE", "eureka"), ("HUGINN_ENDPOINT", "rudp://not-an-address")], "HUGINN_ENDPOINT is not rudp://"),
|
||||
];
|
||||
for (env, expected) in cases {
|
||||
let mut mcp = Mcp::start(env);
|
||||
let (error, whoami) = mcp.call("whoami", json!({}));
|
||||
assert!(!error, "{expected}");
|
||||
assert_eq!(whoami["reachable"], json!(false), "{expected}");
|
||||
assert_eq!(whoami["error"]["error"], json!("Misconfigured"), "{expected}");
|
||||
assert!(whoami["error"]["detail"].as_str().unwrap().contains(expected), "{expected}: {whoami}");
|
||||
for (tool, arguments) in [
|
||||
("view", json!({ "id": absent() })),
|
||||
("query", json!({ "selection": {} })),
|
||||
("admit", faculty_args(vec![])),
|
||||
] {
|
||||
let (error, refusal) = mcp.call(tool, arguments);
|
||||
assert!(error, "{tool}: {expected}");
|
||||
assert_eq!(refusal["error"], json!("Misconfigured"), "{tool}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Input that does not fit the tool's schema is refused before any call is
|
||||
/// made. rmcp 2.2 reports it as a tool result with `isError`, not as a
|
||||
/// JSON-RPC `invalid_params`, so the model sees what to correct. The server
|
||||
/// keeps serving.
|
||||
#[test]
|
||||
fn malformed_input_is_refused_before_any_call() {
|
||||
let server = answering(encode_response("eureka-state-call", "admit", &HuginnMindResponse::View(None), "scripted").unwrap());
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
let response = mcp.rpc("tools/call", json!({ "name": "admit", "arguments": { "faculty": "Hands" } }));
|
||||
assert!(response.get("error").is_none(), "{response}");
|
||||
assert_eq!(response["result"]["isError"], json!(true));
|
||||
let body: Value = serde_json::from_str(response["result"]["content"][0]["text"].as_str().unwrap()).expect("a JSON body");
|
||||
assert_eq!(body["error"], json!("InvalidInput"), "{body}");
|
||||
assert!(body["detail"].as_str().unwrap().contains("missing field"), "{body}");
|
||||
let (error, _) = mcp.call("whoami", json!({}));
|
||||
assert!(!error);
|
||||
}
|
||||
|
||||
/// A request too large for one send is the client's typed refusal, an error
|
||||
/// result that names the size and the limit and does not call the daemon
|
||||
/// unavailable.
|
||||
#[test]
|
||||
fn an_oversize_admit_is_a_typed_too_large_error() {
|
||||
let mut mcp = Mcp::at(INSTANCE, closed_port());
|
||||
let mut arguments = faculty_args(vec![]);
|
||||
arguments["session"] = json!("s".repeat(90_000));
|
||||
let (error, body) = mcp.call("admit", arguments);
|
||||
assert!(error);
|
||||
assert_eq!(body["error"], json!("TooLarge"), "{body}");
|
||||
assert_eq!(body["limit"], json!(eureka_state::MAX_REQUEST_BYTES));
|
||||
assert!(body["bytes"].as_u64().unwrap() > eureka_state::MAX_REQUEST_BYTES as u64, "{body}");
|
||||
}
|
||||
|
||||
/// Who admitted a batch is validated by the mind, whichever door the batch
|
||||
/// came through: a blank, whitespace, multi-line or oversized session or agent
|
||||
/// is a typed refusal, and nothing is committed.
|
||||
#[test]
|
||||
fn admit_refuses_an_unusable_attribution() {
|
||||
let (root, daemon) = mind(vec![vec![stewardship()]]);
|
||||
let server = serve(root, daemon);
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
for (field, value) in [
|
||||
("session", "".to_string()),
|
||||
("session", " ".to_string()),
|
||||
("session", "x".repeat(5_000)),
|
||||
("session", "line\nbreak".to_string()),
|
||||
("agent", "".to_string()),
|
||||
("agent", "two\nlines".to_string()),
|
||||
("agent", "y".repeat(201)),
|
||||
] {
|
||||
let mut arguments = faculty_args(vec![campaign_json()]);
|
||||
arguments[field] = json!(value);
|
||||
let (error, refused) = mcp.call("admit", arguments);
|
||||
assert!(!error, "{field}={value:?}");
|
||||
let refusal = &refused["Refused"]["Document"];
|
||||
let named = json!(format!("provenance.{field}"));
|
||||
assert!(
|
||||
refusal["InvalidFormat"]["field"] == named || refusal["FieldBound"]["field"] == named,
|
||||
"{field}={value:?}: {refused}"
|
||||
);
|
||||
}
|
||||
let (_, whoami) = mcp.call("whoami", json!({}));
|
||||
assert_eq!(whoami["status"]["receipts"], json!(2), "only the seed is committed: {whoami}");
|
||||
let (error, admitted) = mcp.call("admit", faculty_args(vec![campaign_json()]));
|
||||
assert!(!error);
|
||||
assert!(admitted.get("Committed").is_some(), "{admitted}");
|
||||
}
|
||||
|
||||
/// The tool forwards the caller's semantic query, and only that, to the mind.
|
||||
#[test]
|
||||
fn query_sends_the_semantic_request_on_the_wire() {
|
||||
let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let sink = std::sync::Arc::clone(&seen);
|
||||
let server = scripted(move |message| {
|
||||
let (_, request) = huginn_mind::envelope::decode_request(message).expect("a mind request");
|
||||
sink.lock().unwrap().push(request);
|
||||
encode_response("eureka-state-call", "query", &HuginnMindResponse::View(None), "scripted").unwrap()
|
||||
});
|
||||
let mut mcp = Mcp::at(INSTANCE, server.addr);
|
||||
mcp.call("query", json!({ "selection": { "projection": "document" }, "semantic": { "text": "drift", "top_k": 7 } }));
|
||||
mcp.call("query", json!({ "selection": {} }));
|
||||
let seen = seen.lock().unwrap();
|
||||
let [HuginnMindRequest::Query { instance: first, semantic: Some(asked), .. }, HuginnMindRequest::Query { semantic: None, .. }] =
|
||||
&seen[..]
|
||||
else {
|
||||
panic!("two queries expected, the first with its semantic part: {seen:?}");
|
||||
};
|
||||
assert_eq!(first.0, INSTANCE);
|
||||
assert_eq!((asked.text.0.as_str(), asked.top_k), ("drift", 7));
|
||||
}
|
||||
|
||||
/// A daemon that never answers is given up on after the server's own call
|
||||
/// timeout, 15 seconds: not earlier, and not much later.
|
||||
#[test]
|
||||
fn a_silent_daemon_is_given_up_on_after_fifteen_seconds() {
|
||||
let silent = UdpSocket::bind("127.0.0.1:0").unwrap();
|
||||
let mut mcp = Mcp::at(INSTANCE, silent.local_addr().unwrap());
|
||||
let started = std::time::Instant::now();
|
||||
let (error, body) = mcp.call("view", json!({ "id": absent() }));
|
||||
let elapsed = started.elapsed();
|
||||
assert!(error);
|
||||
assert_eq!(body["error"], json!("Unavailable"), "{body}");
|
||||
assert!(body["detail"].as_str().unwrap().contains("15s"), "{body}");
|
||||
assert!(elapsed >= Duration::from_secs(15), "gave up after {elapsed:?}");
|
||||
assert!(elapsed < Duration::from_secs(20), "gave up after {elapsed:?}");
|
||||
}
|
||||
|
||||
/// An instance name that is not UTF-8 is reported as that, not as unset.
|
||||
#[test]
|
||||
fn a_non_utf8_instance_is_reported_as_such() {
|
||||
let mut mcp = Mcp::spawn(|command| {
|
||||
command.env("EUREKA_INSTANCE", non_utf8()).env("HUGINN_ENDPOINT", "rudp://127.0.0.1:1");
|
||||
});
|
||||
let (error, whoami) = mcp.call("whoami", json!({}));
|
||||
assert!(!error);
|
||||
let detail = whoami["error"]["detail"].as_str().unwrap();
|
||||
assert!(detail.contains("EUREKA_INSTANCE is not valid UTF-8"), "{whoami}");
|
||||
assert_eq!(whoami["instance"], Value::Null);
|
||||
}
|
||||
|
||||
/// Likewise an endpoint that is not UTF-8: reported as that, not as unset.
|
||||
#[test]
|
||||
fn a_non_utf8_endpoint_is_reported_as_such() {
|
||||
let mut mcp = Mcp::spawn(|command| {
|
||||
command.env("EUREKA_INSTANCE", INSTANCE).env("HUGINN_ENDPOINT", non_utf8());
|
||||
});
|
||||
let (error, whoami) = mcp.call("whoami", json!({}));
|
||||
assert!(!error);
|
||||
let detail = whoami["error"]["detail"].as_str().unwrap();
|
||||
assert!(detail.contains("HUGINN_ENDPOINT is not valid UTF-8"), "{whoami}");
|
||||
assert_eq!(whoami["endpoint"], Value::Null);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn non_utf8() -> std::ffi::OsString {
|
||||
std::os::unix::ffi::OsStringExt::from_vec(vec![b'e', 0xff])
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn non_utf8() -> std::ffi::OsString {
|
||||
std::os::windows::ffi::OsStringExt::from_wide(&[0x65, 0xd800])
|
||||
}
|
||||
|
||||
/// A well-formed id of a document no mind here holds.
|
||||
fn absent() -> Value {
|
||||
serde_json::to_value(PipelineRef { kind: PipelineKind::CutSpec, id: Short("eureka-state:cut_spec:cut-99.r1".into()) })
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// A client only lists tools from a server that says it has them.
|
||||
#[test]
|
||||
fn initialize_advertises_tools() {
|
||||
let mcp = Mcp::at(INSTANCE, closed_port());
|
||||
assert!(mcp.init["capabilities"]["tools"].is_object(), "{}", mcp.init);
|
||||
}
|
||||
45
crates/huginn-daemon/Cargo.toml
Normal file
45
crates/huginn-daemon/Cargo.toml
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
[package]
|
||||
name = "huginn-daemon"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
publish.workspace = true
|
||||
|
||||
# The organ's CultNet surface: the socket, the sessions, the process lifetime
|
||||
# and the operation envelope, and no rule. Every rule, the instance check, the
|
||||
# receipt, the status and the wire vocabulary are `huginn-mind`'s, and the leaf
|
||||
# and the store type are reached through it so one crate pins one revision.
|
||||
[lib]
|
||||
name = "huginn_daemon"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "huginn-daemon"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
base64 = "0.22"
|
||||
chrono = "0.4.44"
|
||||
cultnet-rs = { git = "https://github.com/GameCult/CultLib.git", rev = "8fc74c709444b884a2bd339e0c306bf9484d99d2", package = "cultnet-rs" }
|
||||
huginn-mind = { path = "../huginn-mind" }
|
||||
rmp-serde = "1"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
# The vector store's and the embedder's HTTP APIs are the xenos boundary, and
|
||||
# JSON exists in `index/ollama.rs` and `index/qdrant.rs` only. `ureq` 3 with
|
||||
# default features off is plain HTTP with no TLS (`rustls` and `gzip` are its
|
||||
# only defaults), which is all the private mesh needs. It stays below 3.4,
|
||||
# whose `base64` 0.23 would put a second `base64` in a tree that has one.
|
||||
serde_json = "1"
|
||||
sha2 = "0.10"
|
||||
signal-hook = "0.3"
|
||||
ureq = { version = "~3.3", default-features = false }
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
|
||||
# Test-only: `Mind::store_path_for`, the grammar-checked door onto a raw
|
||||
# store's path. The daemon's own store-integrity tests reach a store directly
|
||||
# (to break a row on purpose) and must go through the same door `open` does
|
||||
# rather than reconstructing the path by hand (S3, residue batch).
|
||||
huginn-mind = { path = "../huginn-mind", features = ["test-support"] }
|
||||
286
crates/huginn-daemon/src/bodies.rs
Normal file
286
crates/huginn-daemon/src/bodies.rs
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
//! The deferred bodies: answers too large for one send, held for the client to
|
||||
//! fetch chunk by chunk. Ephemeral transport state, never a document store and
|
||||
//! never world truth: it is lost when the process exits, and a client that
|
||||
//! finds a body gone asks the operation again.
|
||||
//!
|
||||
//! Identity is the manifest's `contentHash`, so the same bytes deferred twice
|
||||
//! are one body. Chunking, hashing and the manifest are `cultnet-rs`'s; what is
|
||||
//! owned here is only how long and how much is kept.
|
||||
//!
|
||||
//! A body expires when it has been untouched for the TTL, where retaining it
|
||||
//! again and serving any of its chunks both touch it. When the stored bytes
|
||||
//! exceed the budget the body touched longest ago is evicted first. A chunk
|
||||
//! that two bodies share is stored once and lives until the last of them goes.
|
||||
//! Time is an argument, so nothing here reads a clock; expiry therefore runs
|
||||
//! only when a frame arrives, and an idle daemon holds an expired body until
|
||||
//! the next one.
|
||||
//!
|
||||
//! Limit: there is no per-session or per-peer budget, so any client can evict
|
||||
//! another client's body mid-fetch: four 64 MiB reads flush the 256 MiB store,
|
||||
//! and the loser's next chunk is answered `found: false`. That is acceptable
|
||||
//! for the organ's trusted WireGuard clients, where an instance identity is
|
||||
//! declared attribution and not authentication.
|
||||
|
||||
use std::cell::Cell;
|
||||
use std::collections::{BTreeSet, HashMap};
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use cultnet_rs::{CultMeshCdnArtifactManifest, CultMeshCdnChunk};
|
||||
|
||||
/// The size of one chunk of a deferred body. Four fit one send with room for a
|
||||
/// control-plane answer beside them on the same session.
|
||||
pub const DEFERRED_CHUNK_BYTES: usize = 256 * 1024;
|
||||
/// What the daemon keeps across all deferred bodies.
|
||||
pub const DEFERRED_BUDGET_BYTES: u64 = 256 * 1024 * 1024;
|
||||
|
||||
struct Chunk {
|
||||
bytes: Vec<u8>,
|
||||
/// The content hashes of the retained bodies that reference this chunk.
|
||||
owners: BTreeSet<String>,
|
||||
}
|
||||
|
||||
struct Body {
|
||||
chunk_hashes: BTreeSet<String>,
|
||||
touched_at: Cell<DateTime<Utc>>,
|
||||
/// Orders touches when the clock does not: the largest is the newest.
|
||||
touch_order: Cell<u64>,
|
||||
}
|
||||
|
||||
pub struct DeferredBodies {
|
||||
budget: u64,
|
||||
ttl: chrono::Duration,
|
||||
chunks: HashMap<String, Chunk>,
|
||||
bodies: HashMap<String, Body>,
|
||||
stored: u64,
|
||||
touches: Cell<u64>,
|
||||
}
|
||||
|
||||
impl DeferredBodies {
|
||||
pub fn new(budget: u64, ttl: Duration) -> Self {
|
||||
Self {
|
||||
budget,
|
||||
ttl: chrono::Duration::from_std(ttl).unwrap_or(chrono::Duration::MAX),
|
||||
chunks: HashMap::new(),
|
||||
bodies: HashMap::new(),
|
||||
stored: 0,
|
||||
touches: Cell::new(0),
|
||||
}
|
||||
}
|
||||
|
||||
/// Bytes held: each distinct chunk once.
|
||||
#[cfg(test)]
|
||||
pub fn stored_bytes(&self) -> u64 {
|
||||
self.stored
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn body_count(&self) -> usize {
|
||||
self.bodies.len()
|
||||
}
|
||||
|
||||
/// Holds a body, or, when its bytes are already held, touches it and
|
||||
/// stores nothing. Then evicts, least recently touched first, while the
|
||||
/// stored bytes exceed the budget and another body could be evicted
|
||||
/// instead: the body just retained is the last to go, and one that alone
|
||||
/// exceeds the budget is kept whole.
|
||||
pub fn retain(&mut self, manifest: &CultMeshCdnArtifactManifest, chunks: Vec<CultMeshCdnChunk>, now: DateTime<Utc>) {
|
||||
let content_hash = manifest.content_hash.clone();
|
||||
if let Some(body) = self.bodies.get(&content_hash) {
|
||||
self.touch(body, now);
|
||||
return;
|
||||
}
|
||||
let mut chunk_hashes = BTreeSet::new();
|
||||
for chunk in chunks {
|
||||
chunk_hashes.insert(chunk.chunk_hash.clone());
|
||||
let held = self.chunks.entry(chunk.chunk_hash).or_insert_with(|| {
|
||||
self.stored += chunk.payload.len() as u64;
|
||||
Chunk { bytes: chunk.payload, owners: BTreeSet::new() }
|
||||
});
|
||||
held.owners.insert(content_hash.clone());
|
||||
}
|
||||
let body = Body { chunk_hashes, touched_at: Cell::new(now), touch_order: Cell::new(0) };
|
||||
self.touch(&body, now);
|
||||
self.bodies.insert(content_hash, body);
|
||||
while self.stored > self.budget && self.bodies.len() > 1 {
|
||||
let oldest = self
|
||||
.bodies
|
||||
.iter()
|
||||
.min_by_key(|(_, body)| body.touch_order.get())
|
||||
.map(|(hash, _)| hash.clone())
|
||||
.expect("more than one body is held");
|
||||
self.drop_body(&oldest);
|
||||
}
|
||||
}
|
||||
|
||||
/// One chunk's bytes by its normalised hash, touching every body that
|
||||
/// references it. Serving a chunk is what keeps its body alive.
|
||||
pub fn chunk(&self, hash: &str, now: DateTime<Utc>) -> Option<&[u8]> {
|
||||
let chunk = self.chunks.get(hash)?;
|
||||
for owner in &chunk.owners {
|
||||
self.touch(&self.bodies[owner], now);
|
||||
}
|
||||
Some(&chunk.bytes)
|
||||
}
|
||||
|
||||
/// Drops every body untouched for the TTL as of `now`. A clock that reads
|
||||
/// earlier than a touch expires nothing.
|
||||
pub fn expire(&mut self, now: DateTime<Utc>) {
|
||||
let expired: Vec<String> = self
|
||||
.bodies
|
||||
.iter()
|
||||
.filter(|(_, body)| now.signed_duration_since(body.touched_at.get()) >= self.ttl)
|
||||
.map(|(hash, _)| hash.clone())
|
||||
.collect();
|
||||
for hash in expired {
|
||||
self.drop_body(&hash);
|
||||
}
|
||||
}
|
||||
|
||||
fn touch(&self, body: &Body, now: DateTime<Utc>) {
|
||||
body.touched_at.set(now);
|
||||
self.touches.set(self.touches.get() + 1);
|
||||
body.touch_order.set(self.touches.get());
|
||||
}
|
||||
|
||||
fn drop_body(&mut self, content_hash: &str) {
|
||||
let Some(body) = self.bodies.remove(content_hash) else { return };
|
||||
for hash in body.chunk_hashes {
|
||||
let chunk = self.chunks.get_mut(&hash).expect("a retained body's chunks are held");
|
||||
chunk.owners.remove(content_hash);
|
||||
if chunk.owners.is_empty() {
|
||||
self.stored -= chunk.bytes.len() as u64;
|
||||
self.chunks.remove(&hash);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use cultnet_rs::pack_content;
|
||||
use huginn_mind::MAX_DEFERRED_BODY_BYTES;
|
||||
|
||||
/// The three numbers a client author is told, spelled out: a chunk of
|
||||
/// 256 KiB, a largest deferred body of 64 MiB, a budget of 256 MiB that
|
||||
/// holds at least one whole body.
|
||||
#[test]
|
||||
fn the_deferral_constants_are_the_documented_numbers() {
|
||||
assert_eq!(DEFERRED_CHUNK_BYTES, 262_144);
|
||||
assert_eq!(MAX_DEFERRED_BODY_BYTES, 67_108_864);
|
||||
assert_eq!(DEFERRED_BUDGET_BYTES, 268_435_456);
|
||||
}
|
||||
|
||||
fn at(seconds: i64) -> DateTime<Utc> {
|
||||
DateTime::from_timestamp(1_800_000_000 + seconds, 0).unwrap()
|
||||
}
|
||||
|
||||
/// A body of `pieces` chunks of 4 bytes each, distinct per `tag`.
|
||||
fn body(tag: u8, pieces: usize) -> (CultMeshCdnArtifactManifest, Vec<CultMeshCdnChunk>) {
|
||||
let mut bytes = Vec::new();
|
||||
for piece in 0..pieces {
|
||||
bytes.extend_from_slice(&[tag, piece as u8, 7, 7]);
|
||||
}
|
||||
pack_content("t", "package", "", "", "", &bytes, 4).unwrap()
|
||||
}
|
||||
|
||||
fn held(store: &DeferredBodies, manifest: &CultMeshCdnArtifactManifest, now: DateTime<Utc>) -> bool {
|
||||
manifest.chunks.iter().all(|chunk| store.chunk(&chunk.chunk_hash, now).is_some())
|
||||
}
|
||||
|
||||
const TTL: Duration = Duration::from_secs(60);
|
||||
|
||||
/// The same bytes deferred twice are one body, counted once; serving a
|
||||
/// chunk slides the expiry; an untouched body is gone exactly at the TTL.
|
||||
#[test]
|
||||
fn retention_is_idempotent_and_sliding() {
|
||||
let mut store = DeferredBodies::new(1_000, TTL);
|
||||
let (manifest, chunks) = body(1, 3);
|
||||
store.retain(&manifest, chunks.clone(), at(0));
|
||||
store.retain(&manifest, chunks, at(1));
|
||||
assert_eq!((store.body_count(), store.stored_bytes()), (1, 12), "one body, its bytes counted once");
|
||||
|
||||
// Retaining again at t=1 touched it, so it lives to t=61, not t=60.
|
||||
store.expire(at(60));
|
||||
assert_eq!(store.body_count(), 1, "the second retain refreshed it");
|
||||
// Serving a chunk at t=60 touches it, so t=119 is inside a fresh TTL.
|
||||
assert!(store.chunk(&manifest.chunks[0].chunk_hash, at(60)).is_some());
|
||||
store.expire(at(119));
|
||||
assert_eq!(store.body_count(), 1, "serving a chunk slid the expiry");
|
||||
store.expire(at(120));
|
||||
assert_eq!((store.body_count(), store.stored_bytes()), (0, 0), "untouched for the TTL is gone");
|
||||
|
||||
// An untouched body is gone at the TTL and not before it.
|
||||
let (other, chunks) = body(2, 1);
|
||||
store.retain(&other, chunks, at(200));
|
||||
store.expire(at(259));
|
||||
assert_eq!(store.body_count(), 1);
|
||||
store.expire(at(260));
|
||||
assert_eq!(store.body_count(), 0);
|
||||
}
|
||||
|
||||
/// Time going backwards expires nothing.
|
||||
#[test]
|
||||
fn a_clock_behind_the_last_touch_expires_nothing() {
|
||||
let mut store = DeferredBodies::new(1_000, TTL);
|
||||
let (manifest, chunks) = body(1, 1);
|
||||
store.retain(&manifest, chunks, at(100));
|
||||
store.expire(at(0));
|
||||
assert_eq!(store.body_count(), 1);
|
||||
}
|
||||
|
||||
/// Over the budget the body touched longest ago goes first, not the
|
||||
/// newest and not the first inserted, and a chunk two bodies share stays
|
||||
/// while either lives.
|
||||
#[test]
|
||||
fn eviction_is_least_recently_touched_and_bounded_in_bytes() {
|
||||
// Budget 12 bytes: three chunks of 4.
|
||||
let mut store = DeferredBodies::new(12, TTL);
|
||||
let (a, chunks_a) = body(1, 1);
|
||||
let (b, chunks_b) = body(2, 1);
|
||||
let (c, chunks_c) = body(3, 1);
|
||||
store.retain(&a, chunks_a, at(0));
|
||||
store.retain(&b, chunks_b, at(1));
|
||||
// The oldest-inserted body is touched last, so insertion order and
|
||||
// recency disagree about who goes.
|
||||
assert!(store.chunk(&a.chunks[0].chunk_hash, at(2)).is_some());
|
||||
store.retain(&c, chunks_c, at(3));
|
||||
assert_eq!(store.stored_bytes(), 12);
|
||||
let (d, chunks_d) = body(4, 1);
|
||||
store.retain(&d, chunks_d, at(4));
|
||||
assert_eq!(store.stored_bytes(), 12, "bounded by the budget");
|
||||
assert!(!held(&store, &b, at(5)), "the least recently touched went");
|
||||
assert!(held(&store, &a, at(5)) && held(&store, &c, at(5)) && held(&store, &d, at(5)));
|
||||
|
||||
// A shared chunk survives its first owner.
|
||||
let mut store = DeferredBodies::new(12, TTL);
|
||||
let (x, chunks_x) = body(5, 2); // chunks x0 x1
|
||||
store.retain(&x, chunks_x, at(0));
|
||||
let (y, mut chunks_y) = body(6, 1); // chunk y0, plus x1 shared
|
||||
let mut manifest_y = y.clone();
|
||||
manifest_y.content_hash = "shared-owner".into();
|
||||
chunks_y.push(CultMeshCdnChunk {
|
||||
chunk_hash: x.chunks[1].chunk_hash.clone(),
|
||||
payload: vec![5, 1, 7, 7],
|
||||
});
|
||||
store.retain(&manifest_y, chunks_y, at(1));
|
||||
assert_eq!(store.stored_bytes(), 12, "the shared chunk is stored once");
|
||||
// Evict x (older): its private chunk goes, the shared one stays.
|
||||
let (z, chunks_z) = body(7, 1);
|
||||
store.retain(&z, chunks_z, at(2));
|
||||
assert_eq!(store.body_count(), 2, "x was evicted");
|
||||
assert!(store.chunk(&x.chunks[0].chunk_hash, at(3)).is_none(), "x's private chunk is gone");
|
||||
assert!(store.chunk(&x.chunks[1].chunk_hash, at(3)).is_some(), "the shared chunk lives with y");
|
||||
}
|
||||
|
||||
/// A body that alone exceeds the budget is kept whole rather than
|
||||
/// evicting itself, so a manifest handed out is never a lie.
|
||||
#[test]
|
||||
fn a_body_over_the_budget_by_itself_is_kept() {
|
||||
let mut store = DeferredBodies::new(4, TTL);
|
||||
let (manifest, chunks) = body(1, 3);
|
||||
store.retain(&manifest, chunks, at(0));
|
||||
assert!(held(&store, &manifest, at(0)));
|
||||
}
|
||||
}
|
||||
864
crates/huginn-daemon/src/daemon.rs
Normal file
864
crates/huginn-daemon/src/daemon.rs
Normal file
|
|
@ -0,0 +1,864 @@
|
|||
//! The dispatch: one mind, one request, one response. No socket, no clock, no
|
||||
//! rule. `now` is the caller's, so a test pins it and the loop is the only
|
||||
//! thing that reads a wall clock.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use cultnet_rs::Selection;
|
||||
use huginn_mind::epiphany_pipeline::{PipelineRef, Slug};
|
||||
use huginn_mind::wire::{HuginnMindRequest, HuginnMindResponse, IndexStatus};
|
||||
use huginn_mind::{Mind, MindRefusal, MindStore, PipelineAdmissionOutcome, SemanticQuery};
|
||||
|
||||
/// Where the index attaches. Called after a batch commits, with the refs the
|
||||
/// batch landed, derived writes included; the sink may read each through
|
||||
/// `mind.view`. Its error is logged and never reaches the caller: the index
|
||||
/// does not decide whether a batch was admitted, and the call must not wait on
|
||||
/// the index's transport. `status` is what `whoami` reports of it.
|
||||
///
|
||||
/// A semantic query is asked in two steps because the index answers on its own
|
||||
/// time: `search` starts one and returns a ticket at once, and `searched`
|
||||
/// hands over the searches that have finished since it was last called. The
|
||||
/// candidates are the index's word about nearness and nothing else; the daemon
|
||||
/// judges them through the mind.
|
||||
pub trait IndexSink<S: MindStore> {
|
||||
fn committed(&mut self, mind: &Mind<S>, writes: &[PipelineRef]) -> anyhow::Result<()>;
|
||||
fn status(&self) -> IndexStatus;
|
||||
/// Starts a search for the documents nearest `text`, at most `top_k` of
|
||||
/// them wanted. `Err` says why the index cannot be asked now.
|
||||
fn search(&mut self, text: &str, top_k: u32) -> Result<SearchTicket, String>;
|
||||
/// How long after it was asked a search is worth answering: the index's
|
||||
/// own bound on it, which the loop waiting for the answer must share.
|
||||
fn search_deadline(&self) -> std::time::Duration;
|
||||
/// Every search that finished since the last call, by ticket: its
|
||||
/// candidates and scores, or why it failed.
|
||||
fn searched(&mut self) -> Vec<(SearchTicket, Result<Hits, String>)>;
|
||||
/// The asker has stopped waiting for `ticket`. A search not yet started is
|
||||
/// dropped, so a queue of searches nobody wants cannot hold the index's
|
||||
/// writes off.
|
||||
fn abandon(&mut self, ticket: SearchTicket);
|
||||
}
|
||||
|
||||
/// Names one search the index was asked, so the loop can match the answer to
|
||||
/// the client that is waiting for it.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||
pub struct SearchTicket(pub u64);
|
||||
|
||||
/// What a search returns: candidate documents and how near each scored, best
|
||||
/// first. Nothing else about them.
|
||||
pub type Hits = Vec<(PipelineRef, f32)>;
|
||||
|
||||
/// A semantic query the index has been asked and has not yet answered. The
|
||||
/// serve loop holds it beside the session that asked, and hands it back to
|
||||
/// `Daemon::finish` with the index's answer.
|
||||
pub struct Search {
|
||||
pub ticket: SearchTicket,
|
||||
selection: Selection,
|
||||
semantic: SemanticQuery,
|
||||
}
|
||||
|
||||
/// What a request came to: an answer, or a search the index has yet to
|
||||
/// finish.
|
||||
pub enum Handled {
|
||||
Answered(HuginnMindResponse),
|
||||
Searching(Search),
|
||||
}
|
||||
|
||||
impl Handled {
|
||||
/// The answer, for a request that cannot be a search.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn answered(self) -> HuginnMindResponse {
|
||||
match self {
|
||||
Self::Answered(response) => response,
|
||||
Self::Searching(search) => panic!("expected an answer, got the search {:?}", search.ticket),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What the log says of the hits the mind held no document for: how many, and
|
||||
/// the first few by name (a restored mind can leave hundreds), or nothing when
|
||||
/// the mind held them all.
|
||||
fn dropped_note(dropped: &[PipelineRef]) -> Option<String> {
|
||||
if dropped.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let named: Vec<&str> = dropped.iter().take(5).map(|hit| hit.id.0.as_str()).collect();
|
||||
Some(format!(
|
||||
"huginn: the index returned {} ids this mind does not hold (first {named:?}); they are left out",
|
||||
dropped.len()
|
||||
))
|
||||
}
|
||||
|
||||
/// The envelope's `source_runtime_id` on every response a daemon serving this
|
||||
/// instance sends. Also what the socket is bound under, before a daemon exists.
|
||||
pub fn runtime_id(instance: &Slug) -> String {
|
||||
format!("huginn-{}", instance.0)
|
||||
}
|
||||
|
||||
/// One mind and whatever indexes what lands in it.
|
||||
pub struct Daemon<S: MindStore, I: IndexSink<S>> {
|
||||
mind: Mind<S>,
|
||||
index: I,
|
||||
}
|
||||
|
||||
impl<S: MindStore, I: IndexSink<S>> Daemon<S, I> {
|
||||
/// A daemon over a mind that is already open. `Mind::open` takes the
|
||||
/// store's exclusive lock for the daemon's lifetime, and ruling 15 has it
|
||||
/// refuse before any socket exists: the caller opens the mind first.
|
||||
pub fn new(mind: Mind<S>, index: I) -> Self {
|
||||
Self { mind, index }
|
||||
}
|
||||
|
||||
pub fn with_index<J: IndexSink<S>>(self, index: J) -> Daemon<S, J> {
|
||||
Daemon { mind: self.mind, index }
|
||||
}
|
||||
|
||||
pub fn mind(&self) -> &Mind<S> {
|
||||
&self.mind
|
||||
}
|
||||
|
||||
pub fn index(&self) -> &I {
|
||||
&self.index
|
||||
}
|
||||
|
||||
/// The envelope's `source_runtime_id` on every response this daemon sends.
|
||||
pub fn runtime_id(&self) -> String {
|
||||
runtime_id(self.mind.instance())
|
||||
}
|
||||
|
||||
/// One request, one answer, or a search to finish. `Admit` goes to the mind whole, because the
|
||||
/// batch declares its own instance and admission's A1 is the check. Every
|
||||
/// other request names an instance: `require_instance` now applies the
|
||||
/// leaf's own grammar before it compares identity, so a name outside the
|
||||
/// grammar (a fullwidth fold, say) is refused by name rather than read as
|
||||
/// merely a foreign mind, and the daemon asks nothing twice.
|
||||
pub fn handle(&mut self, request: HuginnMindRequest, now: DateTime<Utc>) -> Handled {
|
||||
if let Some(declared) = request.instance()
|
||||
&& !matches!(request, HuginnMindRequest::Admit(_))
|
||||
&& let Err(refusal) = self.mind.require_instance(declared)
|
||||
{
|
||||
return Handled::Answered(HuginnMindResponse::Refused(refusal));
|
||||
}
|
||||
let answer = match request {
|
||||
HuginnMindRequest::Whoami => HuginnMindResponse::Whoami(self.mind.status(self.index.status())),
|
||||
HuginnMindRequest::Admit(batch) => {
|
||||
let outcome = self.mind.admit(batch, now);
|
||||
if let PipelineAdmissionOutcome::Committed { writes, .. } = &outcome
|
||||
&& let Err(error) = self.index.committed(&self.mind, writes)
|
||||
{
|
||||
let refs = writes.iter().map(|write| write.id.0.clone()).collect::<Vec<_>>().join(", ");
|
||||
eprintln!("huginn: the index refused the landed writes [{refs}]: {error:#}");
|
||||
}
|
||||
HuginnMindResponse::Admit(outcome)
|
||||
}
|
||||
HuginnMindRequest::View { id, .. } => match self.mind.view(&id) {
|
||||
Ok(view) => HuginnMindResponse::View(view),
|
||||
Err(refusal) => HuginnMindResponse::Refused(refusal),
|
||||
},
|
||||
HuginnMindRequest::Query { selection, semantic: None, .. } => match self.mind.query(&selection) {
|
||||
Ok(page) => HuginnMindResponse::Query(page),
|
||||
Err(refusal) => HuginnMindResponse::Refused(refusal),
|
||||
},
|
||||
HuginnMindRequest::Query { selection, semantic: Some(semantic), .. } => {
|
||||
return self.begin_search(selection, semantic);
|
||||
}
|
||||
};
|
||||
Handled::Answered(answer)
|
||||
}
|
||||
|
||||
/// A semantic query starts only if the mind would accept it (so nothing is
|
||||
/// embedded for a query that cannot be ranked) and the index can be asked.
|
||||
fn begin_search(&mut self, selection: Selection, semantic: SemanticQuery) -> Handled {
|
||||
let refused = |refusal: MindRefusal| Handled::Answered(HuginnMindResponse::Refused(refusal));
|
||||
if let Err(refusal) = self.mind.check_semantic(&selection, &semantic) {
|
||||
return refused(refusal);
|
||||
}
|
||||
match self.index.search(&semantic.text.0, semantic.top_k) {
|
||||
Ok(ticket) => Handled::Searching(Search { ticket, selection, semantic }),
|
||||
Err(detail) => refused(MindRefusal::Unavailable { detail }),
|
||||
}
|
||||
}
|
||||
|
||||
/// How long a search is worth waiting for.
|
||||
pub fn search_deadline(&self) -> std::time::Duration {
|
||||
self.index.search_deadline()
|
||||
}
|
||||
|
||||
/// Searches the index has finished since this was last called.
|
||||
pub fn searched(&mut self) -> Vec<(SearchTicket, Result<Hits, String>)> {
|
||||
self.index.searched()
|
||||
}
|
||||
|
||||
/// The client has stopped waiting for `ticket`.
|
||||
pub fn abandon(&mut self, ticket: SearchTicket) {
|
||||
self.index.abandon(ticket);
|
||||
}
|
||||
|
||||
/// The answer to a search: the index's candidates ranked through the mind
|
||||
/// (`Mind::rank` drops what the mind does not hold and lets the selection
|
||||
/// decide the rest), or, when the index could not answer, a typed
|
||||
/// `Unavailable`. Never an empty page in place of a failure.
|
||||
pub fn finish(&mut self, search: Search, found: Result<Hits, String>) -> HuginnMindResponse {
|
||||
let hits = match found {
|
||||
Ok(hits) => hits,
|
||||
Err(detail) => return HuginnMindResponse::Refused(MindRefusal::Unavailable { detail }),
|
||||
};
|
||||
match self.mind.rank(&search.selection, &search.semantic, &hits) {
|
||||
Ok(ranked) => {
|
||||
if let Some(note) = dropped_note(&ranked.dropped) {
|
||||
eprintln!("{note}");
|
||||
}
|
||||
HuginnMindResponse::Query(ranked.page)
|
||||
}
|
||||
Err(refusal) => HuginnMindResponse::Refused(refusal),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod tests {
|
||||
use super::*;
|
||||
use chrono::TimeZone;
|
||||
use huginn_mind::epiphany_pipeline::{
|
||||
AuthorityMap, CodeLocation, CutDelete, CutVerification, Date, DocRef, FileChange, Line, NegativeCheck,
|
||||
OrgRepo, PipelineCampaign, PipelineCutSpec, PipelineDocument, PipelineInstance, PipelineKind,
|
||||
PipelineQuestion, PipelineRuling, PipelineStewardship, QuestionOption, RulingAuthority, Sha, Short,
|
||||
StructuralDelta, Title, VerificationTest,
|
||||
};
|
||||
use huginn_mind::wire::MindStatus;
|
||||
use huginn_mind::{MindRefusal, OwnedRedbMessagePackBackingStore};
|
||||
use std::path::Path;
|
||||
use cultnet_rs::Selection;
|
||||
use huginn_mind::{Faculty, PipelineAdmissionBatch, PipelineProvenance, PipelinePageItems, PipelineStatus};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use tempfile::TempDir;
|
||||
|
||||
pub(crate) const INSTANCE: &str = "yggdrasil";
|
||||
pub(crate) const OTHER: &str = "thought-cage";
|
||||
/// `INSTANCE`'s own length, differing in its last byte alone.
|
||||
pub(crate) const NEAR: &str = "yggdrasix";
|
||||
/// `INSTANCE` whole, with more after it.
|
||||
pub(crate) const PREFIXED: &str = "yggdrasil-two";
|
||||
/// `INSTANCE` in another case, which a slug label permits.
|
||||
pub(crate) const CASED: &str = "Yggdrasil";
|
||||
pub(crate) const CAMPAIGN: &str = "eureka-state";
|
||||
|
||||
pub(crate) fn slug(value: &str) -> Slug {
|
||||
value.into()
|
||||
}
|
||||
|
||||
pub(crate) fn now() -> DateTime<Utc> {
|
||||
Utc.with_ymd_and_hms(2026, 9, 16, 12, 0, 0).unwrap()
|
||||
}
|
||||
|
||||
pub(crate) fn provenance() -> PipelineProvenance {
|
||||
PipelineProvenance {
|
||||
faculty: Faculty::Hands,
|
||||
agent: Short("claude".into()),
|
||||
session: Short("session-1".into()),
|
||||
tool: Short("admit".into()),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn identity(name: &str) -> PipelineDocument {
|
||||
PipelineDocument::Instance(PipelineInstance {
|
||||
instance: slug(name),
|
||||
display_name: Short(format!("{name} mind")),
|
||||
created_at: Date("2026-09-16".into()),
|
||||
host: Short(name.into()),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn batch(instance: &str, documents: Vec<PipelineDocument>) -> PipelineAdmissionBatch {
|
||||
PipelineAdmissionBatch { instance: slug(instance), provenance: provenance(), documents }
|
||||
}
|
||||
|
||||
/// A question and the ruling that answers it: admitting the ruling derives
|
||||
/// the question's resolution, so one batch lands two writes.
|
||||
pub(crate) fn question_and_ruling() -> (PipelineDocument, PipelineDocument) {
|
||||
let question = PipelineDocument::Question(PipelineQuestion {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: "Q1".into(),
|
||||
title: Title("Who owns the state?".into()),
|
||||
question: "Who owns the state?".into(),
|
||||
options: vec![
|
||||
QuestionOption { label: "A".into(), text: "the organ".into() },
|
||||
QuestionOption { label: "B".into(), text: "the client".into() },
|
||||
],
|
||||
recommended: "A".into(),
|
||||
depends: vec![],
|
||||
raised_in: None,
|
||||
asked_on: Date("2026-09-16".into()),
|
||||
});
|
||||
let ruling = PipelineDocument::Ruling(PipelineRuling {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: "R1".into(),
|
||||
title: Title("An instance owns its mind".into()),
|
||||
answers: Some(Short(format!("{CAMPAIGN}:question:Q1"))),
|
||||
choice: Some("A".into()),
|
||||
ruling: "An instance owns its mind.".into(),
|
||||
operator_quote: None,
|
||||
ruled_on: Date("2026-09-16".into()),
|
||||
precedents: vec![],
|
||||
authority: RulingAuthority::Operator,
|
||||
});
|
||||
(question, ruling)
|
||||
}
|
||||
|
||||
pub(crate) const REPO: &str = "GameCult/Epiphany";
|
||||
|
||||
/// A `Line` and a `Short` at the leaf's own bound, so a document built out
|
||||
/// of them is as wide as any writer may make it.
|
||||
pub(crate) fn line(n: usize) -> Line {
|
||||
Line(format!("{n:0>1000}"))
|
||||
}
|
||||
|
||||
pub(crate) fn short(n: usize) -> Short {
|
||||
Short(format!("{n:0>200}"))
|
||||
}
|
||||
|
||||
fn sha() -> Sha {
|
||||
Sha("5f98228d".into())
|
||||
}
|
||||
|
||||
fn doc_ref() -> DocRef {
|
||||
DocRef { path: Short("notes/target.md".into()), start_line: 1, end_line: 9, commit: sha() }
|
||||
}
|
||||
|
||||
/// The three documents a cut spec needs before it may be admitted: the
|
||||
/// mind's identity, its stewardship of the repo, and the campaign that
|
||||
/// names the repo.
|
||||
pub(crate) fn campaign_seed() -> Vec<PipelineDocument> {
|
||||
vec![
|
||||
identity(INSTANCE),
|
||||
PipelineDocument::Stewardship(PipelineStewardship {
|
||||
instance: slug(INSTANCE),
|
||||
repo: OrgRepo(REPO.into()),
|
||||
sequence: 1,
|
||||
assigned_on: Date("2026-09-16".into()),
|
||||
note: "assigned".into(),
|
||||
}),
|
||||
PipelineDocument::Campaign(PipelineCampaign {
|
||||
slug: slug(CAMPAIGN),
|
||||
title: Title("Eureka pipeline state".into()),
|
||||
repos: vec![OrgRepo(REPO.into())],
|
||||
working_branch: Short("codex/eureka-pipeline-state".into()),
|
||||
target_doc: doc_ref(),
|
||||
}),
|
||||
]
|
||||
}
|
||||
|
||||
/// One cut spec with every list the leaf bounds filled to its maximum and
|
||||
/// every text at its own: `file_changes` file changes, 64 deletes, keeps,
|
||||
/// adds, builds, tests, negative checks and operator checks, a full
|
||||
/// authority map and a full structural delta. At 256 file changes that is
|
||||
/// roughly a megabyte of field content, which is what a real-sized answer
|
||||
/// looks like at the leaf's bounds rather than at a fixture's convenience.
|
||||
/// `depends_on`, `rulings` and `questions` stay empty because each is a
|
||||
/// reference admission resolves.
|
||||
pub(crate) fn cut_spec(cut: &str, file_changes: usize) -> PipelineDocument {
|
||||
let lines = |count: usize| (0..count).map(line).collect::<Vec<_>>();
|
||||
let shorts = |count: usize| (0..count).map(short).collect::<Vec<_>>();
|
||||
let location = |n: usize| CodeLocation { path: short(n), line: 1, end_line: Some(9) };
|
||||
PipelineDocument::CutSpec(PipelineCutSpec {
|
||||
campaign: slug(CAMPAIGN),
|
||||
cut: cut.into(),
|
||||
revision: 1,
|
||||
title: Title(short(0).0),
|
||||
repo: OrgRepo(REPO.into()),
|
||||
branch: short(1),
|
||||
base: sha(),
|
||||
depends_on: vec![],
|
||||
first: lines(16),
|
||||
deletes: (0..64).map(|n| CutDelete { path: short(n), lines: 9, note: line(n) }).collect(),
|
||||
keeps_moves: lines(64),
|
||||
adds: lines(64),
|
||||
file_changes: (0..file_changes).map(|n| FileChange { location: location(n), change: line(n) }).collect(),
|
||||
authority_map: Some(AuthorityMap {
|
||||
owner: line(0),
|
||||
inputs: lines(16),
|
||||
outputs: lines(16),
|
||||
derived_state: lines(16),
|
||||
forbidden_writers: lines(16),
|
||||
shared_paths: lines(16),
|
||||
deletion_line: line(1),
|
||||
}),
|
||||
verification: CutVerification {
|
||||
builds: lines(64),
|
||||
tests: (0..64).map(|n| VerificationTest { name: short(n), pins: line(n) }).collect(),
|
||||
negative: (0..64).map(|n| NegativeCheck { pattern: short(n), scope: line(n) }).collect(),
|
||||
operator: lines(64),
|
||||
},
|
||||
estimate: StructuralDelta {
|
||||
lines_added: 0,
|
||||
lines_removed: 0,
|
||||
dependencies_added: shorts(64),
|
||||
dependencies_removed: shorts(64),
|
||||
formats_added: shorts(64),
|
||||
formats_removed: shorts(64),
|
||||
targets_added: shorts(64),
|
||||
targets_removed: shorts(64),
|
||||
},
|
||||
rulings: vec![],
|
||||
questions: vec![],
|
||||
})
|
||||
}
|
||||
|
||||
/// The cut spec at the leaf's own maximum, and one narrow enough that its
|
||||
/// answer still fits a single send. `WIDE_CHANGES` is the leaf's bound;
|
||||
/// `FITTING_CHANGES` is chosen so the view of that spec lands inside the
|
||||
/// window with less than a packet-count's slack, which is what makes the
|
||||
/// window's own value load-bearing rather than merely generous.
|
||||
pub(crate) const WIDE_CUT: &str = "wide";
|
||||
pub(crate) const WIDE_CHANGES: usize = 256;
|
||||
pub(crate) const FITTING_CUT: &str = "fits";
|
||||
pub(crate) const FITTING_CHANGES: usize = 180;
|
||||
|
||||
/// A daemon whose mind holds the campaign seed, one cut spec as wide as the
|
||||
/// leaf permits, and one narrow enough to answer, so a read over it is a
|
||||
/// real-sized answer either way.
|
||||
pub(crate) fn seeded_wide() -> (TempDir, Daemon<OwnedRedbMessagePackBackingStore, NoIndex>) {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut daemon = open_unindexed(root.path(), &slug(INSTANCE)).unwrap();
|
||||
for documents in [
|
||||
campaign_seed(),
|
||||
vec![cut_spec(WIDE_CUT, WIDE_CHANGES), cut_spec(FITTING_CUT, FITTING_CHANGES)],
|
||||
] {
|
||||
let outcome = daemon.handle(HuginnMindRequest::Admit(batch(INSTANCE, documents)), now()).answered();
|
||||
assert!(
|
||||
matches!(outcome, HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { .. })),
|
||||
"{outcome:?}"
|
||||
);
|
||||
}
|
||||
(root, daemon)
|
||||
}
|
||||
|
||||
/// A daemon over a real redb mind in a temporary directory, with its
|
||||
/// identity admitted. The directory is returned because dropping it
|
||||
/// removes the store.
|
||||
pub(crate) fn seeded() -> (TempDir, Daemon<OwnedRedbMessagePackBackingStore, NoIndex>) {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut daemon = open_unindexed(root.path(), &slug(INSTANCE)).unwrap();
|
||||
let outcome = daemon.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![identity(INSTANCE)])), now()).answered();
|
||||
assert!(
|
||||
matches!(outcome, HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { .. })),
|
||||
"{outcome:?}"
|
||||
);
|
||||
(root, daemon)
|
||||
}
|
||||
|
||||
/// The log names how many hits the mind did not hold and the first five, and
|
||||
/// says nothing when it held them all.
|
||||
#[test]
|
||||
fn the_dropped_hits_are_reported_by_count_and_the_first_five_names() {
|
||||
assert_eq!(dropped_note(&[]), None);
|
||||
let dropped: Vec<PipelineRef> =
|
||||
(0..7).map(|n| PipelineRef { kind: PipelineKind::Question, id: Short(format!("gone-{n}")) }).collect();
|
||||
let note = dropped_note(&dropped).unwrap();
|
||||
assert!(note.contains("returned 7 ids"), "{note}");
|
||||
assert!(note.contains("gone-0") && note.contains("gone-4") && !note.contains("gone-5"), "{note}");
|
||||
assert_eq!(dropped_note(&dropped[..1]).unwrap().matches("gone-").count(), 1);
|
||||
}
|
||||
|
||||
/// No index: for the tests whose subject is not the projection.
|
||||
pub(crate) struct NoIndex;
|
||||
|
||||
impl<S: MindStore> IndexSink<S> for NoIndex {
|
||||
fn committed(&mut self, _mind: &Mind<S>, _writes: &[PipelineRef]) -> anyhow::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn status(&self) -> IndexStatus {
|
||||
IndexStatus::Current
|
||||
}
|
||||
|
||||
fn search(&mut self, _text: &str, _top_k: u32) -> Result<SearchTicket, String> {
|
||||
Err("this index cannot search".into())
|
||||
}
|
||||
|
||||
fn search_deadline(&self) -> std::time::Duration {
|
||||
std::time::Duration::ZERO
|
||||
}
|
||||
|
||||
fn searched(&mut self) -> Vec<(SearchTicket, Result<Hits, String>)> {
|
||||
Vec::new()
|
||||
}
|
||||
|
||||
fn abandon(&mut self, _ticket: SearchTicket) {}
|
||||
}
|
||||
|
||||
/// A daemon over the instance's mind under `state_root`, with no index.
|
||||
pub(crate) fn open_unindexed(
|
||||
state_root: &Path,
|
||||
instance: &Slug,
|
||||
) -> Result<Daemon<OwnedRedbMessagePackBackingStore, NoIndex>, MindRefusal> {
|
||||
Ok(Daemon::new(Mind::open(state_root, instance)?, NoIndex))
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct RecordingIndex {
|
||||
seen: Arc<Mutex<Vec<(Vec<String>, Vec<bool>)>>>,
|
||||
}
|
||||
|
||||
impl<S: MindStore> IndexSink<S> for RecordingIndex {
|
||||
fn committed(&mut self, mind: &Mind<S>, writes: &[PipelineRef]) -> anyhow::Result<()> {
|
||||
let ids = writes.iter().map(|write| write.id.0.clone()).collect();
|
||||
let readable = writes.iter().map(|write| mind.view(write).unwrap().is_some()).collect();
|
||||
self.seen.lock().unwrap().push((ids, readable));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn status(&self) -> IndexStatus {
|
||||
IndexStatus::Current
|
||||
}
|
||||
|
||||
fn search(&mut self, _text: &str, _top_k: u32) -> Result<SearchTicket, String> {
|
||||
Err("this index cannot search".into())
|
||||
}
|
||||
|
||||
fn search_deadline(&self) -> std::time::Duration {
|
||||
std::time::Duration::ZERO
|
||||
}
|
||||
|
||||
fn searched(&mut self) -> Vec<(SearchTicket, Result<Hits, String>)> {
|
||||
Vec::new()
|
||||
}
|
||||
|
||||
fn abandon(&mut self, _ticket: SearchTicket) {}
|
||||
}
|
||||
|
||||
struct FailingIndex;
|
||||
|
||||
impl<S: MindStore> IndexSink<S> for FailingIndex {
|
||||
fn committed(&mut self, _mind: &Mind<S>, _writes: &[PipelineRef]) -> anyhow::Result<()> {
|
||||
Err(anyhow::anyhow!("the index is not there"))
|
||||
}
|
||||
|
||||
fn status(&self) -> IndexStatus {
|
||||
IndexStatus::Current
|
||||
}
|
||||
|
||||
fn search(&mut self, _text: &str, _top_k: u32) -> Result<SearchTicket, String> {
|
||||
Err("this index cannot search".into())
|
||||
}
|
||||
|
||||
fn search_deadline(&self) -> std::time::Duration {
|
||||
std::time::Duration::ZERO
|
||||
}
|
||||
|
||||
fn searched(&mut self) -> Vec<(SearchTicket, Result<Hits, String>)> {
|
||||
Vec::new()
|
||||
}
|
||||
|
||||
fn abandon(&mut self, _ticket: SearchTicket) {}
|
||||
}
|
||||
|
||||
fn status(daemon: &mut Daemon<OwnedRedbMessagePackBackingStore, impl IndexSink<OwnedRedbMessagePackBackingStore>>) -> MindStatus {
|
||||
match daemon.handle(HuginnMindRequest::Whoami, now()).answered() {
|
||||
HuginnMindResponse::Whoami(status) => status,
|
||||
other => panic!("expected a status, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Typed hand-off between the organs without a socket: a batch enters and
|
||||
/// every read answers over what it landed, at a clock the test pins.
|
||||
#[test]
|
||||
fn a_batch_round_trips_typed_through_handle_without_a_socket() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut daemon = open_unindexed(root.path(), &slug(INSTANCE)).unwrap();
|
||||
let outcome = daemon.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![identity(INSTANCE)])), now()).answered();
|
||||
let HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { writes, .. }) = outcome else {
|
||||
panic!("expected a commit");
|
||||
};
|
||||
assert_eq!(writes.len(), 1);
|
||||
assert_eq!(writes[0].kind, PipelineKind::Instance);
|
||||
|
||||
assert_eq!(
|
||||
status(&mut daemon),
|
||||
MindStatus {
|
||||
instance: slug(INSTANCE),
|
||||
schema_epoch: "epiphany.pipeline.epoch.v2".into(),
|
||||
documents: 1,
|
||||
receipts: 1,
|
||||
index: IndexStatus::Current,
|
||||
}
|
||||
);
|
||||
assert_eq!(daemon.runtime_id(), "huginn-yggdrasil");
|
||||
|
||||
let instances = Selection { schemas: Some(vec![PipelineKind::Instance.type_id().into()]), ..Selection::default() };
|
||||
let HuginnMindResponse::Query(page) = daemon.handle(
|
||||
HuginnMindRequest::Query { instance: slug(INSTANCE), selection: instances.clone(), semantic: None },
|
||||
now(),
|
||||
).answered() else {
|
||||
panic!("expected a page");
|
||||
};
|
||||
assert_eq!(page.matched, 1);
|
||||
let PipelinePageItems::Headers(headers) = &page.items else { panic!("the default projection is a header") };
|
||||
assert_eq!(headers[0].admission.provenance, provenance());
|
||||
assert_eq!(headers[0].status, huginn_mind::PipelineStatusSummary::InForce);
|
||||
|
||||
let id = writes[0].clone();
|
||||
let whole = Selection { projection: "document".into(), ..instances };
|
||||
let HuginnMindResponse::Query(page) =
|
||||
daemon.handle(HuginnMindRequest::Query { instance: slug(INSTANCE), selection: whole, semantic: None }, now()).answered()
|
||||
else {
|
||||
panic!("expected a page");
|
||||
};
|
||||
let PipelinePageItems::Documents(views) = &page.items else { panic!("the document projection is the view") };
|
||||
assert_eq!(views[0].status, PipelineStatus::InForce);
|
||||
let view = daemon.handle(HuginnMindRequest::View { instance: slug(INSTANCE), id: id.clone() }, now()).answered();
|
||||
assert_eq!(view, HuginnMindResponse::View(Some(views[0].clone())));
|
||||
}
|
||||
|
||||
/// Ruling 14 across the transport, on both sides: the mind refuses a read
|
||||
/// and a write that name another instance, and nothing lands.
|
||||
#[test]
|
||||
fn a_read_or_a_write_naming_another_instance_is_refused_by_the_mind_and_writes_nothing() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut daemon = open_unindexed(root.path(), &slug(INSTANCE)).unwrap();
|
||||
let landed = daemon.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![identity(INSTANCE)])), now()).answered();
|
||||
let HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { writes, .. }) = landed else {
|
||||
panic!("expected a commit");
|
||||
};
|
||||
let foreign = MindRefusal::ForeignInstance { declared: OTHER.into(), mind: INSTANCE.into() };
|
||||
|
||||
let query = HuginnMindRequest::Query { instance: slug(OTHER), selection: Selection::default(), semantic: None };
|
||||
assert_eq!(daemon.handle(query, now()).answered(), HuginnMindResponse::Refused(foreign.clone()));
|
||||
|
||||
let admit = HuginnMindRequest::Admit(batch(OTHER, vec![identity(OTHER)]));
|
||||
assert_eq!(
|
||||
daemon.handle(admit, now()).answered(),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(foreign.clone()))
|
||||
);
|
||||
|
||||
// The declared instance travels to the mind as the client sent it. A
|
||||
// batch whose documents name no instance at all is still refused on
|
||||
// the declaration alone, so nothing between the wire and A1 may
|
||||
// rewrite it into the mind's own.
|
||||
let (question, _) = question_and_ruling();
|
||||
let admit = HuginnMindRequest::Admit(batch(OTHER, vec![question]));
|
||||
assert_eq!(
|
||||
daemon.handle(admit, now()).answered(),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(foreign.clone()))
|
||||
);
|
||||
|
||||
let view = HuginnMindRequest::View { instance: slug(OTHER), id: writes[0].clone() };
|
||||
assert_eq!(daemon.handle(view, now()).answered(), HuginnMindResponse::Refused(foreign));
|
||||
|
||||
// Three names a comparison could mistake for this mind's. `NEAR` is
|
||||
// `INSTANCE`'s own length and differs in its last byte, so a check
|
||||
// reading lengths, first bytes, or running only when the declared name
|
||||
// is longer admits it; `PREFIXED` carries `INSTANCE` whole, so a prefix
|
||||
// test admits it; `CASED` is `INSTANCE` in another case, which a slug
|
||||
// label permits, so a check folding case admits it. Each is refused on
|
||||
// the read side and the write side.
|
||||
assert_eq!(NEAR.len(), INSTANCE.len());
|
||||
assert!(PREFIXED.starts_with(INSTANCE));
|
||||
assert_eq!(CASED.to_ascii_lowercase(), INSTANCE);
|
||||
for declared in [NEAR, PREFIXED, CASED] {
|
||||
let refusal = MindRefusal::ForeignInstance { declared: declared.into(), mind: INSTANCE.into() };
|
||||
let query = HuginnMindRequest::Query { instance: slug(declared), selection: Selection::default(), semantic: None };
|
||||
assert_eq!(daemon.handle(query, now()).answered(), HuginnMindResponse::Refused(refusal.clone()), "{declared}");
|
||||
let view = HuginnMindRequest::View { instance: slug(declared), id: writes[0].clone() };
|
||||
assert_eq!(daemon.handle(view, now()).answered(), HuginnMindResponse::Refused(refusal.clone()), "{declared}");
|
||||
let admit = HuginnMindRequest::Admit(batch(declared, vec![identity(declared)]));
|
||||
assert_eq!(
|
||||
daemon.handle(admit, now()).answered(),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(refusal)),
|
||||
"{declared}"
|
||||
);
|
||||
}
|
||||
|
||||
assert_eq!(status(&mut daemon).documents, 1, "nothing landed");
|
||||
}
|
||||
|
||||
/// A declared instance outside `Slug`'s grammar is refused by its own
|
||||
/// name, `InvalidFormat`, on the read path, before the identity
|
||||
/// comparison ever runs: it is not merely a foreign mind, it names no
|
||||
/// mind a client could compose. A fullwidth Latin small letter y is used
|
||||
/// because it is the shape Soul named: a check that folds it to plain
|
||||
/// ASCII `y` before validating would let this probe pass as grammatical
|
||||
/// and reach `require_instance`, which would then answer `ForeignInstance`
|
||||
/// instead — the wrong refusal, reached the wrong way.
|
||||
#[test]
|
||||
fn a_declared_instance_outside_the_grammar_is_refused_by_name_before_the_identity_check() {
|
||||
let (_root, mut daemon) = seeded();
|
||||
let fullwidth = "\u{FF59}ggdrasil";
|
||||
assert_ne!(fullwidth, INSTANCE, "not the mind's own bytes");
|
||||
assert!(!fullwidth.is_ascii(), "outside the grammar: a Slug is dot-joined ASCII labels");
|
||||
|
||||
let expected = MindRefusal::Document(huginn_mind::epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "declared".into(),
|
||||
value: fullwidth.into(),
|
||||
});
|
||||
|
||||
let query = HuginnMindRequest::Query { instance: slug(fullwidth), selection: Selection::default(), semantic: None };
|
||||
assert_eq!(daemon.handle(query, now()).answered(), HuginnMindResponse::Refused(expected.clone()));
|
||||
|
||||
let view = HuginnMindRequest::View {
|
||||
instance: slug(fullwidth),
|
||||
id: PipelineRef { kind: PipelineKind::Question, id: Short("not-reached".into()) },
|
||||
};
|
||||
assert_eq!(daemon.handle(view, now()).answered(), HuginnMindResponse::Refused(expected));
|
||||
}
|
||||
|
||||
/// `Admit` is excluded from the daemon's own pre-check (its batch is not a
|
||||
/// read naming an instance the mind is asked about first), but it reaches
|
||||
/// the identical `require_instance` from inside `admit_steps`'s A1, so a
|
||||
/// fullwidth *declared batch instance* is `InvalidFormat` there too,
|
||||
/// named and valued the same as the read path's. One door,
|
||||
/// `Mind::require_instance`, decides both. The batch's one document names
|
||||
/// the mind's own real instance, `INSTANCE`, so it clears `admit`'s own
|
||||
/// per-document `Bounded` check and the only fullwidth text anywhere in
|
||||
/// the request is the batch's declared instance, the thing this test
|
||||
/// pins.
|
||||
#[test]
|
||||
fn a_fullwidth_instance_is_refused_the_same_way_by_admit_and_by_query() {
|
||||
let (_root, mut daemon) = seeded();
|
||||
let fullwidth = "\u{FF59}ggdrasil";
|
||||
|
||||
let expected = MindRefusal::Document(huginn_mind::epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "declared".into(),
|
||||
value: fullwidth.into(),
|
||||
});
|
||||
|
||||
let admit = HuginnMindRequest::Admit(batch(fullwidth, vec![identity(INSTANCE)]));
|
||||
assert_eq!(
|
||||
daemon.handle(admit, now()).answered(),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(expected.clone()))
|
||||
);
|
||||
|
||||
let query = HuginnMindRequest::Query { instance: slug(fullwidth), selection: Selection::default(), semantic: None };
|
||||
assert_eq!(daemon.handle(query, now()).answered(), HuginnMindResponse::Refused(expected));
|
||||
}
|
||||
|
||||
/// A refusal a read method raised is the answer the dispatch returns, whole.
|
||||
/// The instance check refuses before the method runs and is pinned above;
|
||||
/// these are the refusals the methods themselves raise, so nothing here is
|
||||
/// the daemon's own. Each must arrive as the mind spelled it: not rewrapped
|
||||
/// as `Unavailable`, not rewrapped as another read's refusal, and not
|
||||
/// swallowed into an empty answer, which would read as "no such document"
|
||||
/// rather than "that is not a reference".
|
||||
#[test]
|
||||
fn a_refusal_a_read_raised_is_the_answer_the_dispatch_returns_whole() {
|
||||
let (_root, mut daemon) = seeded();
|
||||
|
||||
// A reference whose kind and id disagree names no document any writer
|
||||
// could have composed; the leaf's own refusal says so.
|
||||
let invalid = PipelineRef { kind: PipelineKind::Question, id: "not a reference".into() };
|
||||
let malformed = MindRefusal::Document(huginn_mind::epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "ref.id".into(),
|
||||
value: "not a reference".into(),
|
||||
});
|
||||
let view = HuginnMindRequest::View { instance: slug(INSTANCE), id: invalid };
|
||||
assert_eq!(daemon.handle(view, now()).answered(), HuginnMindResponse::Refused(malformed));
|
||||
|
||||
// And the same reads answer rather than refuse when they are asked
|
||||
// something answerable, so the arms above are refusing on the mind's
|
||||
// word and not on their own shape.
|
||||
let absent =
|
||||
PipelineRef { kind: PipelineKind::Question, id: Short(format!("{CAMPAIGN}:question:Q1")) };
|
||||
let present = HuginnMindRequest::View { instance: slug(INSTANCE), id: absent };
|
||||
assert_eq!(daemon.handle(present, now()).answered(), HuginnMindResponse::View(None));
|
||||
let plain = HuginnMindRequest::Query { instance: slug(INSTANCE), selection: Selection::default(), semantic: None };
|
||||
let HuginnMindResponse::Query(page) = daemon.handle(plain, now()).answered() else { panic!("expected a page") };
|
||||
assert_eq!(page.matched, 1);
|
||||
}
|
||||
|
||||
/// A refusal of the selection crosses the dispatch as itself: typed, on
|
||||
/// the field the client sent, and never swallowed into an empty page or
|
||||
/// rewrapped as an unavailable store. A stale cursor is the organ's own
|
||||
/// defect, so a valid walk never meets it.
|
||||
#[test]
|
||||
fn a_selection_refusal_crosses_the_dispatch_as_itself() {
|
||||
let (_root, mut daemon) = seeded();
|
||||
let ask = |daemon: &mut Daemon<OwnedRedbMessagePackBackingStore, NoIndex>, selection: Selection| {
|
||||
daemon.handle(HuginnMindRequest::Query { instance: slug(INSTANCE), selection, semantic: None }, now()).answered()
|
||||
};
|
||||
let predicate = |index: &str, value: &str| Selection {
|
||||
fields: Some(vec![cultnet_rs::FieldPredicate {
|
||||
index: index.into(),
|
||||
op: "any_of".into(),
|
||||
values: Some(vec![value.into()]),
|
||||
number: None,
|
||||
}]),
|
||||
..Selection::default()
|
||||
};
|
||||
let HuginnMindResponse::Refused(MindRefusal::SelectionInvalid { field, .. }) =
|
||||
ask(&mut daemon, predicate("nonsense", "x"))
|
||||
else {
|
||||
panic!("an undeclared alias is the substrate's refusal");
|
||||
};
|
||||
assert_eq!(field, "fields[0].index");
|
||||
let HuginnMindResponse::Refused(MindRefusal::SelectionInvalid { field, value, .. }) =
|
||||
ask(&mut daemon, predicate("repo", "a/b/c"))
|
||||
else {
|
||||
panic!("a value outside its domain is the organ's refusal");
|
||||
};
|
||||
assert_eq!((field.as_str(), value.as_deref()), ("fields[0].values", Some("a/b/c")));
|
||||
let garbled = Selection { cursor: Some("not a cursor".into()), ..Selection::default() };
|
||||
assert!(matches!(ask(&mut daemon, garbled), HuginnMindResponse::Refused(MindRefusal::CursorInvalid { .. })));
|
||||
assert_eq!(ask(&mut daemon, predicate("repo", "a/b/c")).status(), "rejected");
|
||||
}
|
||||
|
||||
/// Cut 11's seam: the index is handed every landed write, derived ones
|
||||
/// included, and can read each; an index that fails changes nothing.
|
||||
#[test]
|
||||
fn the_index_is_handed_every_landed_write_after_commit_and_never_decides_the_outcome() {
|
||||
let (_root, daemon) = seeded();
|
||||
let recording = RecordingIndex::default();
|
||||
let mut daemon = daemon.with_index(recording.clone());
|
||||
let (question, ruling) = question_and_ruling();
|
||||
|
||||
let outcome = daemon.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![question, ruling])), now()).answered();
|
||||
let HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { writes, .. }) = outcome else {
|
||||
panic!("expected a commit");
|
||||
};
|
||||
let landed: Vec<String> = writes.iter().map(|write| write.id.0.clone()).collect();
|
||||
assert_eq!(landed.len(), 3, "the ruling answers the question, so a resolution is derived: {landed:?}");
|
||||
|
||||
let seen = recording.seen.lock().unwrap().clone();
|
||||
assert_eq!(seen.len(), 1);
|
||||
assert_eq!(seen[0].0, landed, "the sink sees exactly the landed writes");
|
||||
assert_eq!(seen[0].1, vec![true, true, true], "and can read each one when it is called");
|
||||
|
||||
let (question, ruling) = question_and_ruling();
|
||||
let mut daemon = daemon.with_index(FailingIndex);
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut fresh = open_unindexed(root.path(), &slug(INSTANCE)).unwrap().with_index(FailingIndex);
|
||||
fresh.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![identity(INSTANCE)])), now()).answered();
|
||||
let outcome = fresh.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![question, ruling])), now()).answered();
|
||||
let HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { writes, .. }) = outcome else {
|
||||
panic!("a failing index does not refuse the batch");
|
||||
};
|
||||
for write in &writes {
|
||||
let view = fresh.handle(HuginnMindRequest::View { instance: slug(INSTANCE), id: write.clone() }, now()).answered();
|
||||
assert!(matches!(view, HuginnMindResponse::View(Some(_))), "{write:?} is readable");
|
||||
}
|
||||
assert_eq!(status(&mut daemon).documents, 4, "identity, question, ruling and the derived resolution");
|
||||
}
|
||||
|
||||
/// Ruling 15: a mind that will not open is refused, and opening it takes
|
||||
/// no address, so no socket can precede it. `startup`'s order is
|
||||
/// pinned in `serve::tests`.
|
||||
#[test]
|
||||
fn the_daemon_refuses_loudly_when_it_cannot_open_the_mind_and_binds_nothing() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let held = open_unindexed(root.path(), &slug(INSTANCE)).unwrap();
|
||||
let path = Mind::<OwnedRedbMessagePackBackingStore>::store_path_for(root.path(), &slug(INSTANCE)).unwrap();
|
||||
assert_eq!(
|
||||
open_unindexed(root.path(), &slug(INSTANCE)).err(),
|
||||
Some(MindRefusal::MindAlreadyOwned { path: path.display().to_string() })
|
||||
);
|
||||
drop(held);
|
||||
|
||||
let mut planted = open_unindexed(root.path(), &slug(INSTANCE)).unwrap();
|
||||
planted.handle(HuginnMindRequest::Admit(batch(INSTANCE, vec![identity(INSTANCE)])), now()).answered();
|
||||
drop(planted);
|
||||
let moved = Mind::<OwnedRedbMessagePackBackingStore>::store_path_for(root.path(), &slug(OTHER)).unwrap();
|
||||
std::fs::create_dir_all(moved.parent().unwrap()).unwrap();
|
||||
std::fs::copy(&path, &moved).unwrap();
|
||||
assert_eq!(
|
||||
open_unindexed(root.path(), &slug(OTHER)).err(),
|
||||
Some(MindRefusal::ForeignInstance { declared: OTHER.into(), mind: INSTANCE.into() })
|
||||
);
|
||||
}
|
||||
}
|
||||
239
crates/huginn-daemon/src/index/fakes.rs
Normal file
239
crates/huginn-daemon/src/index/fakes.rs
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
//! In-memory ports for the projector's tests. Each fake is a handle on shared
|
||||
//! state, so a test keeps observing after it has moved the fake into a
|
||||
//! projector or a worker thread.
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::sync::{Arc, Condvar, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Result, bail, ensure};
|
||||
|
||||
use super::{CollectionMeta, Described, Embedder, Hit, ModelIdentity, Point, VectorIndex};
|
||||
|
||||
pub(crate) fn identity(digest: &str) -> ModelIdentity {
|
||||
ModelIdentity { name: "model".into(), digest: digest.into(), dimensions: 4 }
|
||||
}
|
||||
|
||||
/// A door a test holds shut: `embed` announces it has arrived and waits.
|
||||
#[derive(Clone, Default)]
|
||||
pub(crate) struct Gate(Arc<(Mutex<(u32, bool)>, Condvar)>);
|
||||
|
||||
impl Gate {
|
||||
pub(crate) fn shut() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Blocks until `count` calls to `embed` have arrived at the gate.
|
||||
pub(crate) fn wait_arrived(&self, count: u32) {
|
||||
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
|
||||
let mut state = self.0.0.lock().unwrap();
|
||||
while state.0 < count {
|
||||
let left = deadline.saturating_duration_since(std::time::Instant::now());
|
||||
assert!(!left.is_zero(), "{count} calls never arrived at the gate; {} did", state.0);
|
||||
state = self.0.1.wait_timeout(state, left).unwrap().0;
|
||||
}
|
||||
}
|
||||
|
||||
/// Shuts the door again and forgets who arrived, so a test that let start-up
|
||||
/// work through can hold the next call.
|
||||
pub(crate) fn shut_again(&self) {
|
||||
let mut state = self.0.0.lock().unwrap();
|
||||
*state = (0, false);
|
||||
}
|
||||
|
||||
pub(crate) fn open(&self) {
|
||||
self.0.0.lock().unwrap().1 = true;
|
||||
self.0.1.notify_all();
|
||||
}
|
||||
|
||||
fn pass(&self) {
|
||||
let mut state = self.0.0.lock().unwrap();
|
||||
state.0 += 1;
|
||||
self.0.1.notify_all();
|
||||
while !state.1 {
|
||||
state = self.0.1.wait(state).unwrap();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) struct EmbedderState {
|
||||
pub(crate) identity: ModelIdentity,
|
||||
pub(crate) down: bool,
|
||||
/// The texts of every completed call, in order.
|
||||
pub(crate) embedded: Vec<Vec<String>>,
|
||||
/// The bound each completed call was given, in order.
|
||||
pub(crate) bounds: Vec<Duration>,
|
||||
/// The length of the vectors it answers with, where that is not the
|
||||
/// identity's.
|
||||
pub(crate) vector_length: Option<usize>,
|
||||
/// The bound each identity read was given, in order.
|
||||
pub(crate) identity_bounds: Vec<Duration>,
|
||||
/// Whether an identity read is never answered: it fails when its bound
|
||||
/// has passed, as an adapter that honours its bound does.
|
||||
pub(crate) hang_identity: bool,
|
||||
/// How long an identity read takes to answer, when it answers.
|
||||
pub(crate) identity_delay: Duration,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct FakeEmbedder {
|
||||
pub(crate) state: Arc<Mutex<EmbedderState>>,
|
||||
gate: Option<Gate>,
|
||||
}
|
||||
|
||||
impl FakeEmbedder {
|
||||
pub(crate) fn new(digest: &str) -> Self {
|
||||
let state = EmbedderState { identity: identity(digest), down: false, embedded: Vec::new(), bounds: Vec::new(), vector_length: None, identity_bounds: Vec::new(), hang_identity: false, identity_delay: Duration::ZERO };
|
||||
Self { state: Arc::new(Mutex::new(state)), gate: None }
|
||||
}
|
||||
|
||||
pub(crate) fn behind(mut self, gate: &Gate) -> Self {
|
||||
self.gate = Some(gate.clone());
|
||||
self
|
||||
}
|
||||
|
||||
pub(crate) fn texts(&self) -> Vec<String> {
|
||||
self.state.lock().unwrap().embedded.iter().flatten().cloned().collect()
|
||||
}
|
||||
}
|
||||
|
||||
impl Embedder for FakeEmbedder {
|
||||
fn model_identity(&mut self, within: Duration) -> Result<ModelIdentity> {
|
||||
let mut state = self.state.lock().unwrap();
|
||||
state.identity_bounds.push(within);
|
||||
if state.hang_identity {
|
||||
drop(state);
|
||||
std::thread::sleep(within);
|
||||
bail!("the embedder did not answer within {within:?}");
|
||||
}
|
||||
if state.down {
|
||||
bail!("the embedder is down");
|
||||
}
|
||||
let (identity, delay) = (state.identity.clone(), state.identity_delay);
|
||||
drop(state);
|
||||
std::thread::sleep(delay);
|
||||
Ok(identity)
|
||||
}
|
||||
|
||||
fn embed(&mut self, texts: &[String], within: Duration) -> Result<Vec<Vec<f32>>> {
|
||||
if let Some(gate) = &self.gate {
|
||||
gate.pass();
|
||||
}
|
||||
let mut state = self.state.lock().unwrap();
|
||||
if state.down {
|
||||
bail!("the embedder is down");
|
||||
}
|
||||
state.embedded.push(texts.to_vec());
|
||||
state.bounds.push(within);
|
||||
let dimensions = state.vector_length.unwrap_or(state.identity.dimensions as usize);
|
||||
Ok(texts.iter().map(|text| (0..dimensions).map(|at| (text.len() + at) as f32).collect()).collect())
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) struct Stored {
|
||||
pub(crate) label: Described,
|
||||
pub(crate) points: BTreeMap<String, Point>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
pub(crate) struct IndexState {
|
||||
pub(crate) collections: BTreeMap<String, Stored>,
|
||||
pub(crate) down: bool,
|
||||
pub(crate) recreated: Vec<(String, CollectionMeta)>,
|
||||
/// The point ids of every upsert call, in order.
|
||||
pub(crate) upserts: Vec<Vec<String>>,
|
||||
/// What every search answers, whatever the vector: the test decides the
|
||||
/// candidates, as a real index would decide them from vectors.
|
||||
pub(crate) hits: Vec<Hit>,
|
||||
/// The collection, vector length and limit of every search, in order.
|
||||
pub(crate) searches: Vec<(String, usize, u32)>,
|
||||
/// How many times every point id was listed: only a reconciliation does.
|
||||
pub(crate) listed: u32,
|
||||
/// Whether a search fails at the vector store, the rest of it working.
|
||||
pub(crate) fail_search: bool,
|
||||
/// The bound each describe and each search was given, in order, by call.
|
||||
pub(crate) bounds: Vec<(&'static str, Duration)>,
|
||||
/// How many points the collection held when each search reached it.
|
||||
pub(crate) held_when_searched: Vec<usize>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
pub(crate) struct FakeIndex {
|
||||
pub(crate) state: Arc<Mutex<IndexState>>,
|
||||
}
|
||||
|
||||
impl FakeIndex {
|
||||
pub(crate) fn holding(&self, collection: &str) -> BTreeSet<String> {
|
||||
let state = self.state.lock().unwrap();
|
||||
state.collections.get(collection).map(|stored| stored.points.keys().cloned().collect()).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Plants a collection this organ did not make, or made for another mind.
|
||||
pub(crate) fn plant(&self, collection: &str, label: Described) {
|
||||
self.state.lock().unwrap().collections.insert(collection.into(), Stored { label, points: BTreeMap::new() });
|
||||
}
|
||||
}
|
||||
|
||||
impl VectorIndex for FakeIndex {
|
||||
fn describe(&mut self, collection: &str, within: Duration) -> Result<Described> {
|
||||
let mut state = self.state.lock().unwrap();
|
||||
state.bounds.push(("describe", within));
|
||||
if state.down {
|
||||
bail!("the vector store is down");
|
||||
}
|
||||
Ok(state.collections.get(collection).map_or(Described::Absent, |stored| stored.label.clone()))
|
||||
}
|
||||
|
||||
fn recreate(&mut self, collection: &str, meta: &CollectionMeta) -> Result<()> {
|
||||
let mut state = self.state.lock().unwrap();
|
||||
if state.down {
|
||||
bail!("the vector store is down");
|
||||
}
|
||||
state.recreated.push((collection.into(), meta.clone()));
|
||||
state
|
||||
.collections
|
||||
.insert(collection.into(), Stored { label: Described::Labelled(meta.clone()), points: BTreeMap::new() });
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn ids(&mut self, collection: &str) -> Result<BTreeSet<String>> {
|
||||
let mut state = self.state.lock().unwrap();
|
||||
if state.down {
|
||||
bail!("the vector store is down");
|
||||
}
|
||||
state.listed += 1;
|
||||
Ok(state.collections.get(collection).map(|stored| stored.points.keys().cloned().collect()).unwrap_or_default())
|
||||
}
|
||||
|
||||
fn upsert(&mut self, collection: &str, points: &[Point]) -> Result<()> {
|
||||
let mut state = self.state.lock().unwrap();
|
||||
if state.down {
|
||||
bail!("the vector store is down");
|
||||
}
|
||||
state.upserts.push(points.iter().map(|point| point.id.clone()).collect());
|
||||
let Some(stored) = state.collections.get_mut(collection) else { bail!("no collection {collection}") };
|
||||
for point in points {
|
||||
stored.points.insert(point.id.clone(), point.clone());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn search(&mut self, collection: &str, vector: &[f32], limit: u32, within: Duration) -> Result<Vec<Hit>> {
|
||||
let mut state = self.state.lock().unwrap();
|
||||
state.bounds.push(("search", within));
|
||||
if state.down {
|
||||
bail!("the vector store is down");
|
||||
}
|
||||
state.searches.push((collection.into(), vector.len(), limit));
|
||||
ensure!(!state.fail_search, "the vector store failed the search");
|
||||
ensure!(state.collections.contains_key(collection), "no collection {collection}");
|
||||
let held = state.collections[collection].points.len();
|
||||
state.held_when_searched.push(held);
|
||||
Ok(state.hits.clone())
|
||||
}
|
||||
}
|
||||
|
||||
/// A working embedder and vector store, for tests whose subject is elsewhere.
|
||||
pub(crate) fn pair() -> (FakeEmbedder, FakeIndex) {
|
||||
(FakeEmbedder::new("d1"), FakeIndex::default())
|
||||
}
|
||||
1095
crates/huginn-daemon/src/index/mod.rs
Normal file
1095
crates/huginn-daemon/src/index/mod.rs
Normal file
File diff suppressed because it is too large
Load diff
75
crates/huginn-daemon/src/index/ollama.rs
Normal file
75
crates/huginn-daemon/src/index/ollama.rs
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
//! The embedder port over Ollama's HTTP API. This file and `qdrant.rs` are the
|
||||
//! xenos boundary: JSON exists here and nowhere else in the organ.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use super::{Embedder, Http, Method, ModelIdentity};
|
||||
|
||||
/// The instruction Qwen3 embedding models take in front of a query (never in
|
||||
/// front of a document). Cut 11b's semantic read prepends it as
|
||||
/// `Instruct: <this>\nQuery: <text>`.
|
||||
pub const QUERY_INSTRUCTION: &str =
|
||||
"Given a question about a project's pipeline state, retrieve the documents that answer it";
|
||||
|
||||
/// One model on one Ollama. Every call carries its own bound.
|
||||
pub struct OllamaEmbedder {
|
||||
base_url: String,
|
||||
model: String,
|
||||
}
|
||||
|
||||
impl OllamaEmbedder {
|
||||
pub fn new(base_url: &str, model: &str) -> Self {
|
||||
Self { base_url: base_url.to_owned(), model: model.to_owned() }
|
||||
}
|
||||
|
||||
fn ask(&self, http: &Http, method: Method, path: &str, body: Option<&Value>) -> Result<Value> {
|
||||
let text = body.map(Value::to_string);
|
||||
let (status, answer) = http.request(method, path, text.as_deref())?;
|
||||
ensure!(status == 200, "Ollama answered {status} to {path}: {}", answer.chars().take(300).collect::<String>());
|
||||
serde_json::from_str(&answer).with_context(|| format!("Ollama's answer to {path} is not JSON"))
|
||||
}
|
||||
}
|
||||
|
||||
impl Embedder for OllamaEmbedder {
|
||||
/// The configured name, the digest Ollama lists for it, and the model's
|
||||
/// embedding length. The address is not part of the identity.
|
||||
fn model_identity(&mut self, within: Duration) -> Result<ModelIdentity> {
|
||||
let http = Http::new(&self.base_url, within);
|
||||
let tags = self.ask(&http, Method::Get, "/api/tags", None)?;
|
||||
let listed = tags["models"].as_array().context("Ollama's tag list has no models")?;
|
||||
let digest = listed
|
||||
.iter()
|
||||
.find(|entry| entry["name"] == self.model.as_str() || entry["model"] == self.model.as_str())
|
||||
.and_then(|entry| entry["digest"].as_str())
|
||||
.with_context(|| format!("Ollama does not list the model {}", self.model))?
|
||||
.to_owned();
|
||||
let shown = self.ask(&http, Method::Post, "/api/show", Some(&json!({ "model": self.model })))?;
|
||||
let info = shown["model_info"].as_object().context("Ollama's model description has no model_info")?;
|
||||
let Some(length) = info.iter().find(|(key, _)| key.ends_with(".embedding_length")).and_then(|(_, value)| value.as_u64())
|
||||
else {
|
||||
bail!("Ollama does not say how long {}'s embeddings are", self.model);
|
||||
};
|
||||
let dimensions = u32::try_from(length).context("an embedding length beyond u32")?;
|
||||
Ok(ModelIdentity { name: self.model.clone(), digest, dimensions })
|
||||
}
|
||||
|
||||
fn embed(&mut self, texts: &[String], within: Duration) -> Result<Vec<Vec<f32>>> {
|
||||
let bounded = Http::new(&self.base_url, within);
|
||||
let answer = self.ask(&bounded, Method::Post, "/api/embed", Some(&json!({ "model": self.model, "input": texts })))?;
|
||||
let embeddings = answer["embeddings"].as_array().context("Ollama's answer has no embeddings")?;
|
||||
embeddings
|
||||
.iter()
|
||||
.map(|vector| {
|
||||
vector
|
||||
.as_array()
|
||||
.context("an embedding is not a list")?
|
||||
.iter()
|
||||
.map(|number| number.as_f64().map(|value| value as f32).context("an embedding holds a non-number"))
|
||||
.collect()
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
158
crates/huginn-daemon/src/index/qdrant.rs
Normal file
158
crates/huginn-daemon/src/index/qdrant.rs
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
//! The vector-store port over Qdrant's HTTP API. With `ollama.rs`, the xenos
|
||||
//! boundary: JSON exists here and nowhere else in the organ.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use super::{CollectionMeta, INDEX_CALL_TIMEOUT, Described, Hit, Http, Method, Point, VectorIndex};
|
||||
|
||||
/// One Qdrant. Every path names a collection this organ was told to own; it
|
||||
/// never lists or touches another.
|
||||
pub struct QdrantIndex {
|
||||
http: Http,
|
||||
}
|
||||
|
||||
/// The ids fetched per scroll page.
|
||||
const SCROLL_PAGE: u64 = 1000;
|
||||
|
||||
impl QdrantIndex {
|
||||
pub fn new(base_url: &str) -> Self {
|
||||
Self { http: Http::new(base_url, INDEX_CALL_TIMEOUT) }
|
||||
}
|
||||
|
||||
fn ask(&self, method: Method, path: &str, body: Option<&Value>, ok: &[u16]) -> Result<(u16, Value)> {
|
||||
self.ask_with(&self.http, method, path, body, ok)
|
||||
}
|
||||
|
||||
fn ask_with(&self, http: &Http, method: Method, path: &str, body: Option<&Value>, ok: &[u16]) -> Result<(u16, Value)> {
|
||||
let text = body.map(Value::to_string);
|
||||
let (status, answer) = http.request(method, path, text.as_deref())?;
|
||||
ensure!(ok.contains(&status), "Qdrant answered {status} to {path}: {}", answer.chars().take(300).collect::<String>());
|
||||
let parsed = if answer.is_empty() { Value::Null } else { serde_json::from_str(&answer).unwrap_or(Value::Null) };
|
||||
Ok((status, parsed))
|
||||
}
|
||||
}
|
||||
|
||||
fn metadata_json(meta: &CollectionMeta) -> Value {
|
||||
json!({
|
||||
"managed_by": meta.managed_by,
|
||||
"instance": meta.instance,
|
||||
"mind": meta.mind,
|
||||
"model": meta.model,
|
||||
"model_digest": meta.model_digest,
|
||||
"dimensions": meta.dimensions,
|
||||
"index_text_version": meta.index_text_version,
|
||||
})
|
||||
}
|
||||
|
||||
/// What a collection's `config.metadata` says. A collection whose metadata
|
||||
/// does not name a writer and an instance is unlabelled; one that does but is
|
||||
/// missing the rest reads as an empty mind or model, which will not match: an
|
||||
/// empty mind is refused as another's, an empty model is rebuilt when the
|
||||
/// writer, the instance and the mind are ours.
|
||||
fn described(metadata: &Value) -> Described {
|
||||
let text = |key: &str| metadata[key].as_str().map(str::to_owned);
|
||||
let (Some(managed_by), Some(instance)) = (text("managed_by"), text("instance")) else {
|
||||
return Described::Unlabelled;
|
||||
};
|
||||
let number = |key: &str| metadata[key].as_u64().and_then(|value| u32::try_from(value).ok()).unwrap_or(0);
|
||||
Described::Labelled(CollectionMeta {
|
||||
managed_by,
|
||||
instance,
|
||||
mind: text("mind").unwrap_or_default(),
|
||||
model: text("model").unwrap_or_default(),
|
||||
model_digest: text("model_digest").unwrap_or_default(),
|
||||
dimensions: number("dimensions"),
|
||||
index_text_version: number("index_text_version"),
|
||||
})
|
||||
}
|
||||
|
||||
impl VectorIndex for QdrantIndex {
|
||||
fn describe(&mut self, collection: &str, within: Duration) -> Result<Described> {
|
||||
let http = self.http.bounded(within);
|
||||
let (status, answer) = self.ask_with(&http, Method::Get, &format!("/collections/{collection}"), None, &[200, 404])?;
|
||||
if status == 404 {
|
||||
return Ok(Described::Absent);
|
||||
}
|
||||
Ok(described(&answer["result"]["config"]["metadata"]))
|
||||
}
|
||||
|
||||
fn recreate(&mut self, collection: &str, meta: &CollectionMeta) -> Result<()> {
|
||||
let path = format!("/collections/{collection}");
|
||||
self.ask(Method::Delete, &format!("{path}?wait=true"), None, &[200, 404])?;
|
||||
let body = json!({
|
||||
"vectors": { "size": meta.dimensions, "distance": "Cosine", "on_disk": true },
|
||||
"metadata": metadata_json(meta),
|
||||
});
|
||||
self.ask(Method::Put, &format!("{path}?wait=true"), Some(&body), &[200])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn ids(&mut self, collection: &str) -> Result<BTreeSet<String>> {
|
||||
let mut ids = BTreeSet::new();
|
||||
let mut offset = Value::Null;
|
||||
loop {
|
||||
let mut body = json!({ "limit": SCROLL_PAGE, "with_payload": false, "with_vector": false });
|
||||
if !offset.is_null() {
|
||||
body["offset"] = offset;
|
||||
}
|
||||
let (_, answer) =
|
||||
self.ask(Method::Post, &format!("/collections/{collection}/points/scroll"), Some(&body), &[200])?;
|
||||
let points = answer["result"]["points"].as_array().context("Qdrant's scroll has no points")?;
|
||||
for point in points {
|
||||
ids.insert(point["id"].as_str().context("a point id is not a string")?.to_owned());
|
||||
}
|
||||
offset = answer["result"]["next_page_offset"].clone();
|
||||
if offset.is_null() {
|
||||
return Ok(ids);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn upsert(&mut self, collection: &str, points: &[Point]) -> Result<()> {
|
||||
let points: Vec<Value> = points
|
||||
.iter()
|
||||
.map(|point| {
|
||||
json!({
|
||||
"id": point.id,
|
||||
"vector": point.vector,
|
||||
"payload": {
|
||||
"doc_id": point.payload.doc_id,
|
||||
"kind": point.payload.kind,
|
||||
"root": point.payload.root,
|
||||
"ordinal": point.payload.ordinal,
|
||||
"text_sha256": point.payload.text_sha256,
|
||||
},
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
self.ask(
|
||||
Method::Put,
|
||||
&format!("/collections/{collection}/points?wait=true"),
|
||||
Some(&json!({ "points": points })),
|
||||
&[200],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn search(&mut self, collection: &str, vector: &[f32], limit: u32, within: Duration) -> Result<Vec<Hit>> {
|
||||
let http = self.http.bounded(within);
|
||||
let body = json!({ "vector": vector, "limit": limit, "with_payload": ["doc_id", "kind"], "with_vector": false });
|
||||
let (_, answer) =
|
||||
self.ask_with(&http, Method::Post, &format!("/collections/{collection}/points/search"), Some(&body), &[200])?;
|
||||
let found = answer["result"].as_array().context("Qdrant's search has no result")?;
|
||||
found
|
||||
.iter()
|
||||
.map(|point| {
|
||||
Ok(Hit {
|
||||
doc_id: point["payload"]["doc_id"].as_str().context("a hit carries no doc_id")?.to_owned(),
|
||||
kind: point["payload"]["kind"].as_str().context("a hit carries no kind")?.to_owned(),
|
||||
score: point["score"].as_f64().context("a hit carries no score")? as f32,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
1948
crates/huginn-daemon/src/index/tests.rs
Normal file
1948
crates/huginn-daemon/src/index/tests.rs
Normal file
File diff suppressed because it is too large
Load diff
25
crates/huginn-daemon/src/lib.rs
Normal file
25
crates/huginn-daemon/src/lib.rs
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
//! `huginn-daemon`: one process that serves one mind over CultNet.
|
||||
//!
|
||||
//! The crate owns four things and no rule. `startup` owns "may this process
|
||||
//! serve this mind", by delegating to `Mind::open` and refusing before
|
||||
//! anything listens. `index` owns when the mind's documents are embedded and
|
||||
//! written to the vector store, off the serving path. `Daemon::handle` owns "which `Mind` method answers this
|
||||
//! request": it dispatches, it asks `Mind::require_instance` for every read
|
||||
//! that names an instance, and it compares nothing itself. `serve` owns "which
|
||||
//! session gets which reply" and whether an answer is delivered whole, deferred or refused; `bodies`
|
||||
//! owns how long and how much of a deferred answer is held.
|
||||
//!
|
||||
//! Every admission rule, the instance check, the receipt, the derived status
|
||||
//! and the wire vocabulary are `huginn-mind`'s. The leaf and the store type
|
||||
//! are reached through `huginn_mind`, so one crate pins one revision of each.
|
||||
|
||||
pub mod bodies;
|
||||
pub mod daemon;
|
||||
pub mod index;
|
||||
pub mod serve;
|
||||
|
||||
#[cfg(test)]
|
||||
mod semantic_tests;
|
||||
|
||||
pub use daemon::{Daemon, Handled, Hits, IndexSink, Search, SearchTicket, runtime_id};
|
||||
pub use serve::{Options, ServeOptions, parse_options, run, startup};
|
||||
37
crates/huginn-daemon/src/main.rs
Normal file
37
crates/huginn-daemon/src/main.rs
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
//! The process: open one instance's mind, bind one socket, serve until a stop
|
||||
//! signal. A mind that will not open ends the process with the refusal on
|
||||
//! stderr and no socket ever bound.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::AtomicBool;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use huginn_daemon::index::Backoff;
|
||||
use huginn_daemon::index::ollama::OllamaEmbedder;
|
||||
use huginn_daemon::index::qdrant::QdrantIndex;
|
||||
use huginn_daemon::serve::{ServeOptions, parse_options, run, startup};
|
||||
|
||||
fn main() -> Result<()> {
|
||||
let options = parse_options(std::env::args().skip(1))?;
|
||||
let embedder = OllamaEmbedder::new(&options.ollama_url, &options.embedding_model);
|
||||
let index = QdrantIndex::new(&options.qdrant_url);
|
||||
let (mut daemon, mut hub, registry) = match startup(&options, embedder, index, Backoff::default()) {
|
||||
Ok(started) => started,
|
||||
Err(error) => {
|
||||
eprintln!("huginn-daemon refuses to serve {}: {error:#}", options.instance.0);
|
||||
std::process::exit(1);
|
||||
}
|
||||
};
|
||||
|
||||
// Idunn runs this process as PID 1 of its own PID namespace, and a
|
||||
// namespace init ignores every signal it has not caught. Without this,
|
||||
// a stop is a ninety-second wait for SIGKILL.
|
||||
let stopping = Arc::new(AtomicBool::new(false));
|
||||
for signal in [signal_hook::consts::SIGTERM, signal_hook::consts::SIGINT] {
|
||||
signal_hook::flag::register(signal, Arc::clone(&stopping))
|
||||
.context("registering Huginn's stop signal handler")?;
|
||||
}
|
||||
|
||||
eprintln!("huginn-daemon serving {} on {}", options.instance.0, hub.local_addr()?);
|
||||
run(&mut daemon, &mut hub, ®istry, &stopping, &ServeOptions::default())
|
||||
}
|
||||
491
crates/huginn-daemon/src/semantic_tests.rs
Normal file
491
crates/huginn-daemon/src/semantic_tests.rs
Normal file
|
|
@ -0,0 +1,491 @@
|
|||
//! The semantic read through the real serve loop: a UDP hub, real sessions, the
|
||||
//! worker thread, and fake embedder and vector-store ports whose behaviour the
|
||||
//! test decides. Nothing here calls `Daemon::handle` directly: the rules under
|
||||
//! test are about what the loop does while an answer is outstanding.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::thread::JoinHandle;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::Result;
|
||||
use cultnet_rs::{CULTNET_OPERATION_CONNECTION_ID, CultMesh, CultMeshRudpSocketOptions, FieldPredicate, Selection};
|
||||
use huginn_mind::epiphany_pipeline::{
|
||||
Date, PipelineDocument, PipelineKind, PipelineQuestion, PipelineResolution, PipelineRef, PipelineRuling,
|
||||
QuestionOption, ResolutionOutcome, RulingAuthority, Short, Title,
|
||||
};
|
||||
use huginn_mind::wire::{HuginnMindRequest, HuginnMindResponse, IndexStatus};
|
||||
use huginn_mind::{Mind, MindRefusal, OwnedRedbMessagePackBackingStore, PipelineAdmissionOutcome, PipelinePageItems, SemanticQuery};
|
||||
use tempfile::TempDir;
|
||||
|
||||
use crate::daemon::Daemon;
|
||||
use crate::daemon::tests::{CAMPAIGN, INSTANCE, batch, campaign_seed, now, slug};
|
||||
use huginn_mind::envelope::{decode_response, encode_request};
|
||||
use crate::index::fakes::{FakeEmbedder, FakeIndex, Gate};
|
||||
use crate::index::{Backoff, Described, Hit, WorkerSink, collection_name};
|
||||
use crate::serve::{ServeOptions, bind, run, schema_registry};
|
||||
|
||||
type Client = cultnet_rs::CultNetRudpSocketTransportConnection;
|
||||
|
||||
fn question(label: &str) -> PipelineDocument {
|
||||
PipelineDocument::Question(PipelineQuestion {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: label.into(),
|
||||
title: Title(format!("Question {label}")),
|
||||
question: format!("Question {label}?").as_str().into(),
|
||||
options: vec![
|
||||
QuestionOption { label: "A".into(), text: "the organ".into() },
|
||||
QuestionOption { label: "B".into(), text: "the client".into() },
|
||||
],
|
||||
recommended: "A".into(),
|
||||
depends: vec![],
|
||||
raised_in: None,
|
||||
asked_on: Date("2026-09-16".into()),
|
||||
})
|
||||
}
|
||||
|
||||
/// A ruling, answering a question when it names one.
|
||||
fn ruling(label: &str, answers: Option<&str>) -> PipelineDocument {
|
||||
PipelineDocument::Ruling(PipelineRuling {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: label.into(),
|
||||
title: Title(format!("Ruling {label}")),
|
||||
answers: answers.map(|question| Short(id("question", question))),
|
||||
choice: answers.map(|_| "A".into()),
|
||||
ruling: format!("Ruling {label}.").as_str().into(),
|
||||
operator_quote: None,
|
||||
ruled_on: Date("2026-09-16".into()),
|
||||
precedents: vec![],
|
||||
authority: RulingAuthority::Operator,
|
||||
})
|
||||
}
|
||||
|
||||
fn resolution(subject: PipelineKind, key: &str, outcome: ResolutionOutcome) -> PipelineDocument {
|
||||
PipelineDocument::Resolution(PipelineResolution {
|
||||
subject: PipelineRef { kind: subject, id: Short(key.into()) },
|
||||
sequence: 1,
|
||||
outcome,
|
||||
rationale: "Resolved.".into(),
|
||||
resolved_on: Date("2026-09-16".into()),
|
||||
})
|
||||
}
|
||||
|
||||
fn id(kind: &str, local: &str) -> String {
|
||||
format!("{CAMPAIGN}:{kind}:{local}")
|
||||
}
|
||||
|
||||
fn hit(kind: &str, key: String, score: f32) -> Hit {
|
||||
Hit { doc_id: key, kind: kind.into(), score }
|
||||
}
|
||||
|
||||
/// Every way a document stops standing, and two that do: Q1 is answered (so
|
||||
/// resolved), Q2 was answered and the answer withdrawn (its resolution is the
|
||||
/// resolved one and the question stands again), R3 is superseded by R4.
|
||||
fn shelf() -> Vec<Vec<PipelineDocument>> {
|
||||
vec![
|
||||
campaign_seed(),
|
||||
vec![question("Q1"), question("Q2")],
|
||||
vec![ruling("R1", Some("Q1"))],
|
||||
vec![ruling("R2", Some("Q2"))],
|
||||
vec![resolution(PipelineKind::Resolution, &id("resolution", "question.Q2.n1"), ResolutionOutcome::Withdrawn { reason: "moot".into() })],
|
||||
vec![ruling("R3", None), ruling("R4", None)],
|
||||
vec![resolution(
|
||||
PipelineKind::Ruling,
|
||||
&id("ruling", "R3"),
|
||||
ResolutionOutcome::Superseded { by: vec![PipelineRef { kind: PipelineKind::Ruling, id: Short(id("ruling", "R4")) }] },
|
||||
)],
|
||||
]
|
||||
}
|
||||
|
||||
fn standing() -> Selection {
|
||||
Selection {
|
||||
fields: Some(vec![FieldPredicate {
|
||||
index: "in_force".into(),
|
||||
op: "any_of".into(),
|
||||
values: Some(vec!["true".into()]),
|
||||
number: None,
|
||||
}]),
|
||||
..Selection::default()
|
||||
}
|
||||
}
|
||||
|
||||
fn semantic(selection: Selection, top_k: u32) -> HuginnMindRequest {
|
||||
semantic_of("who owns the state", selection, top_k)
|
||||
}
|
||||
|
||||
fn semantic_of(text: &str, selection: Selection, top_k: u32) -> HuginnMindRequest {
|
||||
HuginnMindRequest::Query {
|
||||
instance: slug(INSTANCE),
|
||||
selection,
|
||||
semantic: Some(SemanticQuery { text: text.into(), top_k }),
|
||||
}
|
||||
}
|
||||
|
||||
/// How many documents a plain query finds.
|
||||
fn documents(client: &mut Client, message_id: &str) -> usize {
|
||||
ids(&ask(client, message_id, &plain())).len()
|
||||
}
|
||||
|
||||
fn plain() -> HuginnMindRequest {
|
||||
HuginnMindRequest::Query { instance: slug(INSTANCE), selection: Selection::default(), semantic: None }
|
||||
}
|
||||
|
||||
/// The ids on a page, in its order.
|
||||
fn ids(response: &HuginnMindResponse) -> Vec<String> {
|
||||
let HuginnMindResponse::Query(page) = response else { panic!("expected a page, got {response:?}") };
|
||||
match &page.items {
|
||||
PipelinePageItems::Headers(headers) => headers.iter().map(|header| header.id.id.0.clone()).collect(),
|
||||
PipelinePageItems::Documents(views) => views.iter().map(|view| view.id.id.0.clone()).collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn quick() -> Backoff {
|
||||
Backoff { initial: Duration::from_millis(1), max: Duration::from_millis(2), recheck: Duration::from_secs(3600), search_deadline: Duration::from_secs(30) }
|
||||
}
|
||||
|
||||
/// A daemon serving a mind over a real socket, with the ports the test gave it.
|
||||
struct Harness {
|
||||
endpoint: String,
|
||||
stopping: Arc<AtomicBool>,
|
||||
serving: Option<JoinHandle<Result<()>>>,
|
||||
embedder: FakeEmbedder,
|
||||
index: FakeIndex,
|
||||
_root: TempDir,
|
||||
}
|
||||
|
||||
impl Harness {
|
||||
fn start(seed: Vec<Vec<PipelineDocument>>, embedder: FakeEmbedder, index: FakeIndex, search_timeout: Duration) -> Self {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut mind = Mind::open(root.path(), &slug(INSTANCE)).unwrap();
|
||||
for documents in seed {
|
||||
let outcome = mind.admit(batch(INSTANCE, documents), now());
|
||||
assert!(matches!(outcome, PipelineAdmissionOutcome::Committed { .. }), "{outcome:?}");
|
||||
}
|
||||
let sink = WorkerSink::spawn(
|
||||
embedder.clone(),
|
||||
index.clone(),
|
||||
&slug(INSTANCE),
|
||||
mind.genesis_receipt_id().unwrap(),
|
||||
mind.index_entries(None).unwrap(),
|
||||
Backoff { search_deadline: search_timeout, ..quick() },
|
||||
);
|
||||
let mut daemon: Daemon<OwnedRedbMessagePackBackingStore, WorkerSink> = Daemon::new(mind, sink);
|
||||
let mut hub = bind("127.0.0.1:0".parse().unwrap(), &daemon.runtime_id()).unwrap();
|
||||
let endpoint = format!("rudp://{}", hub.local_addr().unwrap());
|
||||
let registry = schema_registry().unwrap();
|
||||
let stopping = Arc::new(AtomicBool::new(false));
|
||||
let loop_stopping = Arc::clone(&stopping);
|
||||
let options = ServeOptions::default();
|
||||
let serving = std::thread::spawn(move || run(&mut daemon, &mut hub, ®istry, &loop_stopping, &options));
|
||||
Self { endpoint, stopping, serving: Some(serving), embedder, index, _root: root }
|
||||
}
|
||||
|
||||
fn client(&self) -> Client {
|
||||
let mut client = CultMesh::create_rudp_client_for_endpoint(
|
||||
"semantic-test".to_string(),
|
||||
CULTNET_OPERATION_CONNECTION_ID,
|
||||
&self.endpoint,
|
||||
CultMeshRudpSocketOptions::default(),
|
||||
)
|
||||
.unwrap();
|
||||
client.connect(Vec::new()).unwrap();
|
||||
for _ in 0..1000 {
|
||||
let _ = client.receive_once();
|
||||
if client.connected() {
|
||||
return client;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
panic!("the client never connected");
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Harness {
|
||||
fn drop(&mut self) {
|
||||
self.stopping.store(true, Ordering::Relaxed);
|
||||
// A loop that is stuck waiting on the index is a failure of the test,
|
||||
// not a reason to hang the run: give it a moment and leave it behind.
|
||||
if let Some(serving) = self.serving.take() {
|
||||
for _ in 0..1500 {
|
||||
if serving.is_finished() {
|
||||
let _ = serving.join();
|
||||
return;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn send(client: &mut Client, message_id: &str, request: &HuginnMindRequest) {
|
||||
client.send_schema_message(&encode_request(message_id, request, None).unwrap()).unwrap();
|
||||
}
|
||||
|
||||
/// The reply to `message_id` on this client's session, or `None` if none came
|
||||
/// in `patience` polls of two milliseconds.
|
||||
fn receive(client: &mut Client, message_id: &str, patience: u32) -> Option<HuginnMindResponse> {
|
||||
for _ in 0..patience {
|
||||
client.poll_resends().unwrap();
|
||||
if let Some(reply) = client.receive_schema_message_once().unwrap() {
|
||||
let (correlated, response) = decode_response(&reply).unwrap();
|
||||
assert_eq!(correlated, message_id);
|
||||
return Some(response.expect("an answer, not an envelope failure"));
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn ask(client: &mut Client, message_id: &str, request: &HuginnMindRequest) -> HuginnMindResponse {
|
||||
send(client, message_id, request);
|
||||
receive(client, message_id, 3000).unwrap_or_else(|| panic!("no reply to {message_id}"))
|
||||
}
|
||||
|
||||
fn index_status(client: &mut Client, message_id: &str) -> IndexStatus {
|
||||
match ask(client, message_id, &HuginnMindRequest::Whoami) {
|
||||
HuginnMindResponse::Whoami(status) => status.index,
|
||||
other => panic!("expected a status, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
fn until_index(client: &mut Client, what: &str, predicate: impl Fn(&IndexStatus) -> bool) {
|
||||
for at in 0..2000 {
|
||||
if predicate(&index_status(client, &format!("status-{at}"))) {
|
||||
return;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(5));
|
||||
}
|
||||
panic!("the index never reported {what}");
|
||||
}
|
||||
|
||||
fn healthy() -> Harness {
|
||||
let (embedder, index) = (FakeEmbedder::new("d1"), FakeIndex::default());
|
||||
Harness::start(shelf(), embedder, index, Duration::from_secs(30))
|
||||
}
|
||||
|
||||
/// A semantic answer is the mind's: the index names a resolved question, a
|
||||
/// withdrawn resolution, a superseded ruling and a document the mind does not
|
||||
/// hold, all scoring above the two that stand, and only the two that stand come
|
||||
/// back, in score order. The selection's other predicates still cut the
|
||||
/// candidates, and the embedder saw the query behind its instruction while the
|
||||
/// index was asked for an oversample.
|
||||
#[test]
|
||||
fn a_semantic_query_returns_only_documents_the_mind_holds_and_the_selection_admits() {
|
||||
let harness = healthy();
|
||||
let mut client = harness.client();
|
||||
until_index(&mut client, "Current", |status| *status == IndexStatus::Current);
|
||||
harness.index.state.lock().unwrap().hits = vec![
|
||||
hit("question", id("question", "Q1"), 0.95),
|
||||
hit("resolution", id("resolution", "question.Q2.n1"), 0.9),
|
||||
hit("ruling", id("ruling", "R3"), 0.85),
|
||||
hit("question", id("question", "Q99"), 0.8),
|
||||
hit("ruling", id("ruling", "R4"), 0.5),
|
||||
hit("question", id("question", "Q2"), 0.4),
|
||||
];
|
||||
|
||||
let standing_only = ask(&mut client, "m-1", &semantic(standing(), 5));
|
||||
assert_eq!(ids(&standing_only), vec![id("ruling", "R4"), id("question", "Q2")]);
|
||||
let HuginnMindResponse::Query(page) = &standing_only else { unreachable!() };
|
||||
assert_eq!((page.matched, page.next.as_ref()), (2, None));
|
||||
|
||||
let rulings = Selection { schemas: Some(vec![PipelineKind::Ruling.type_id().into()]), ..standing() };
|
||||
assert_eq!(ids(&ask(&mut client, "m-2", &semantic(rulings, 5))), vec![id("ruling", "R4")], "the selection's schemas still apply");
|
||||
|
||||
let everything = ids(&ask(&mut client, "m-3", &semantic(Selection::default(), 5)));
|
||||
assert!(everything.contains(&id("question", "Q1")) && !everything.contains(&id("question", "Q99")),
|
||||
"a hit the selection does not exclude is shown, and one the mind does not hold never is: {everything:?}");
|
||||
|
||||
let embedded = harness.embedder.texts();
|
||||
assert!(embedded.iter().any(|text| text.starts_with("Instruct: ") && text.ends_with("\nQuery: who owns the state")), "{embedded:?}");
|
||||
let searches = harness.index.state.lock().unwrap().searches.clone();
|
||||
assert!(searches.iter().all(|(name, _, limit)| *name == collection_name(&slug(INSTANCE)) && *limit == 20), "{searches:?}");
|
||||
}
|
||||
|
||||
/// A cursor cannot resume a ranked answer and a ranked answer has no order to
|
||||
/// reverse: each is refused by name, and nothing is embedded or searched for it.
|
||||
#[test]
|
||||
fn a_cursor_or_a_reversal_with_a_semantic_query_is_refused_and_asks_the_index_nothing() {
|
||||
let harness = healthy();
|
||||
let mut client = harness.client();
|
||||
until_index(&mut client, "Current", |status| *status == IndexStatus::Current);
|
||||
let embedded = harness.embedder.texts().len();
|
||||
|
||||
let with_cursor = Selection { cursor: Some("anything".into()), ..standing() };
|
||||
let refusal = ask(&mut client, "m-1", &semantic(with_cursor, 5));
|
||||
assert!(
|
||||
matches!(&refusal, HuginnMindResponse::Refused(MindRefusal::SelectionInvalid { field, .. }) if field == "cursor"),
|
||||
"{refusal:?}"
|
||||
);
|
||||
assert_eq!(harness.embedder.texts().len(), embedded);
|
||||
assert!(harness.index.state.lock().unwrap().searches.is_empty());
|
||||
|
||||
let reversed = Selection { descending: true, ..standing() };
|
||||
let refusal = ask(&mut client, "m-2", &semantic(reversed, 5));
|
||||
assert!(
|
||||
matches!(&refusal, HuginnMindResponse::Refused(MindRefusal::SelectionInvalid { field, .. }) if field == "descending"),
|
||||
"{refusal:?}"
|
||||
);
|
||||
assert_eq!(harness.embedder.texts().len(), embedded);
|
||||
assert!(harness.index.state.lock().unwrap().searches.is_empty());
|
||||
}
|
||||
|
||||
/// The loop never waits on the network. The embedder is shut, the worker is
|
||||
/// inside it with one client's semantic query, and a second client's `whoami`
|
||||
/// and plain query are answered at once; the first is answered when the gate
|
||||
/// opens. The gate counts arrivals, so this observes where the work is and not
|
||||
/// how long it took.
|
||||
#[test]
|
||||
fn another_session_is_answered_while_a_semantic_search_is_outstanding() {
|
||||
let gate = Gate::shut();
|
||||
gate.open();
|
||||
let embedder = FakeEmbedder::new("d1").behind(&gate);
|
||||
let harness = Harness::start(shelf(), embedder, FakeIndex::default(), Duration::from_secs(30));
|
||||
let mut asker = harness.client();
|
||||
let mut second = harness.client();
|
||||
let mut other = harness.client();
|
||||
until_index(&mut other, "Current", |status| *status == IndexStatus::Current);
|
||||
let held = documents(&mut other, "m-count");
|
||||
harness.index.state.lock().unwrap().hits = vec![hit("ruling", id("ruling", "R4"), 0.5)];
|
||||
|
||||
gate.shut_again();
|
||||
send(&mut asker, "m-ask", &semantic(standing(), 5));
|
||||
gate.wait_arrived(1);
|
||||
assert!(receive(&mut asker, "m-ask", 20).is_none(), "the question is not answered while its embedding is outstanding");
|
||||
// A second question waits behind the first, and each is answered to its own asker.
|
||||
send(&mut second, "m-ask-2", &semantic(Selection { schemas: Some(vec![PipelineKind::Question.type_id().into()]), ..Selection::default() }, 5));
|
||||
|
||||
assert!(matches!(ask(&mut other, "m-who", &HuginnMindRequest::Whoami), HuginnMindResponse::Whoami(_)));
|
||||
assert_eq!(documents(&mut other, "m-plain"), held, "a plain query is answered too");
|
||||
|
||||
gate.open();
|
||||
let answered = receive(&mut asker, "m-ask", 3000).expect("the search is answered once the embedder returns");
|
||||
assert_eq!(ids(&answered), vec![id("ruling", "R4")]);
|
||||
let second_answer = receive(&mut second, "m-ask-2", 3000).expect("the second search is answered too, on its own session");
|
||||
assert_eq!(ids(&second_answer), Vec::<String>::new(), "the index offered a ruling, which its selection of questions does not admit");
|
||||
}
|
||||
|
||||
/// A search the index never finishes is answered at the deadline the client
|
||||
/// was promised, typed and naming the wait, never as an empty page.
|
||||
#[test]
|
||||
fn a_search_the_index_never_finishes_is_answered_unavailable_at_the_deadline() {
|
||||
let gate = Gate::shut();
|
||||
gate.open();
|
||||
let embedder = FakeEmbedder::new("d1").behind(&gate);
|
||||
let harness = Harness::start(shelf(), embedder, FakeIndex::default(), Duration::from_millis(150));
|
||||
let mut client = harness.client();
|
||||
until_index(&mut client, "Current", |status| *status == IndexStatus::Current);
|
||||
|
||||
gate.shut_again();
|
||||
let answered = ask(&mut client, "m-late", &semantic(standing(), 5));
|
||||
let HuginnMindResponse::Refused(MindRefusal::Unavailable { detail }) = &answered else { panic!("{answered:?}") };
|
||||
assert!(detail.contains("did not finish"), "{detail}");
|
||||
gate.open();
|
||||
assert!(matches!(ask(&mut client, "m-after", &HuginnMindRequest::Whoami), HuginnMindResponse::Whoami(_)), "the late answer is dropped, not misdelivered");
|
||||
}
|
||||
|
||||
/// An index that cannot answer is a typed `Unavailable` and never an empty
|
||||
/// page, in each way it cannot: the embedder is down when the query arrives;
|
||||
/// the worker is failing; the collection belongs to someone else. Plain
|
||||
/// queries are unaffected throughout.
|
||||
#[test]
|
||||
fn an_index_that_cannot_answer_is_unavailable_and_plain_queries_still_work() {
|
||||
let unavailable = |response: HuginnMindResponse| match response {
|
||||
HuginnMindResponse::Refused(MindRefusal::Unavailable { detail }) => detail,
|
||||
other => panic!("expected Unavailable, got {other:?}"),
|
||||
};
|
||||
|
||||
// The embedder goes down after the index was Current: the query itself fails.
|
||||
let harness = healthy();
|
||||
let mut client = harness.client();
|
||||
until_index(&mut client, "Current", |status| *status == IndexStatus::Current);
|
||||
let held = documents(&mut client, "m-count");
|
||||
harness.embedder.state.lock().unwrap().down = true;
|
||||
let detail = unavailable(ask(&mut client, "m-down", &semantic(standing(), 5)));
|
||||
assert!(detail.contains("the embedder is down"), "{detail}");
|
||||
assert_eq!(documents(&mut client, "m-plain"), held);
|
||||
|
||||
// A write while it is down makes the worker Failing, and the next query is
|
||||
// refused without being asked of it.
|
||||
let admit = HuginnMindRequest::Admit(batch(INSTANCE, vec![question("Q7")]));
|
||||
assert!(matches!(ask(&mut client, "m-admit", &admit), HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed { .. })));
|
||||
until_index(&mut client, "Failing", |status| matches!(status, IndexStatus::Failing { .. }));
|
||||
let searches = harness.index.state.lock().unwrap().searches.len();
|
||||
let detail = unavailable(ask(&mut client, "m-failing", &semantic(standing(), 5)));
|
||||
assert!(detail.contains("failing") && detail.contains("the embedder is down"), "{detail}");
|
||||
assert_eq!(harness.index.state.lock().unwrap().searches.len(), searches);
|
||||
assert_eq!(documents(&mut client, "m-plain-2"), held + 1, "the admitted question is in the mind whatever the index says");
|
||||
|
||||
// A collection that is another writer's is Refused, and so are queries.
|
||||
let (embedder, index) = (FakeEmbedder::new("d1"), FakeIndex::default());
|
||||
index.plant(&collection_name(&slug(INSTANCE)), Described::Unlabelled);
|
||||
let refused = Harness::start(shelf(), embedder, index, Duration::from_secs(30));
|
||||
let mut client = refused.client();
|
||||
until_index(&mut client, "Refused", |status| matches!(status, IndexStatus::Refused { .. }));
|
||||
let detail = unavailable(ask(&mut client, "m-refused", &semantic(standing(), 5)));
|
||||
assert!(detail.contains("refused") && detail.contains("no Huginn metadata"), "{detail}");
|
||||
assert_eq!(documents(&mut client, "m-plain-3"), held);
|
||||
}
|
||||
|
||||
/// A search the client has stopped waiting for is not started. The embedder is
|
||||
/// held with the first search inside it; a second search queues behind it and
|
||||
/// times out, and when the embedder is released the worker embeds the first,
|
||||
/// skips the second and embeds a third: the index is never asked for an answer
|
||||
/// nobody is waiting for.
|
||||
#[test]
|
||||
fn a_search_that_timed_out_while_queued_is_never_embedded() {
|
||||
let gate = Gate::shut();
|
||||
gate.open();
|
||||
let embedder = FakeEmbedder::new("d1").behind(&gate);
|
||||
let harness = Harness::start(shelf(), embedder, FakeIndex::default(), Duration::from_millis(300));
|
||||
let mut first = harness.client();
|
||||
let mut second = harness.client();
|
||||
let mut third = harness.client();
|
||||
until_index(&mut third, "Current", |status| *status == IndexStatus::Current);
|
||||
let queries = |harness: &Harness| -> Vec<String> {
|
||||
harness.embedder.texts().into_iter().filter(|text| text.starts_with("Instruct: ")).collect()
|
||||
};
|
||||
|
||||
gate.shut_again();
|
||||
send(&mut first, "m-first", &semantic_of("first", standing(), 5));
|
||||
gate.wait_arrived(1);
|
||||
send(&mut second, "m-second", &semantic_of("second", standing(), 5));
|
||||
let timed_out = receive(&mut second, "m-second", 3000).expect("the second search is answered at its deadline");
|
||||
assert!(matches!(&timed_out, HuginnMindResponse::Refused(MindRefusal::Unavailable { detail }) if detail.contains("did not finish")), "{timed_out:?}");
|
||||
|
||||
gate.open();
|
||||
send(&mut third, "m-third", &semantic_of("third", standing(), 5));
|
||||
let answered = receive(&mut third, "m-third", 3000).expect("the third search is answered");
|
||||
assert!(matches!(answered, HuginnMindResponse::Query(_)), "{answered:?}");
|
||||
let embedded = queries(&harness);
|
||||
assert_eq!(embedded.len(), 2, "the first and the third were embedded; the second was taken back before it started: {embedded:?}");
|
||||
assert!(embedded[0].ends_with("Query: first") && embedded[1].ends_with("Query: third"), "{embedded:?}");
|
||||
}
|
||||
|
||||
/// A search asked while the index lags the mind is not refused: it waits behind
|
||||
/// the writes ahead of it, through the real serve loop, and reads them. The
|
||||
/// worker is inside its first batch with the rest of the mind still to write;
|
||||
/// the reply does not come until the gate opens, and it comes with every
|
||||
/// document already in the collection.
|
||||
#[test]
|
||||
fn a_search_asked_while_the_index_lags_waits_for_the_writes_ahead_of_it_and_reads_them() {
|
||||
let gate = Gate::shut();
|
||||
let embedder = FakeEmbedder::new("d1").behind(&gate);
|
||||
let harness = Harness::start(shelf(), embedder, FakeIndex::default(), Duration::from_secs(30));
|
||||
gate.wait_arrived(1);
|
||||
let mut client = harness.client();
|
||||
let IndexStatus::Behind { pending } = index_status(&mut client, "m-status") else { panic!("the worker is inside its first batch") };
|
||||
assert!(pending > 0);
|
||||
harness.index.state.lock().unwrap().hits = vec![hit("ruling", id("ruling", "R4"), 0.5)];
|
||||
|
||||
send(&mut client, "m-behind", &semantic(standing(), 5));
|
||||
assert!(receive(&mut client, "m-behind", 100).is_none(), "the search waits: it is neither refused nor answered from a half-filled collection");
|
||||
|
||||
gate.open();
|
||||
let answered = receive(&mut client, "m-behind", 3000).expect("the search is answered once the index has caught up");
|
||||
assert_eq!(ids(&answered), vec![id("ruling", "R4")]);
|
||||
until_index(&mut client, "Current", |status| *status == IndexStatus::Current);
|
||||
let state = harness.index.state.lock().unwrap();
|
||||
let written = state.collections[&collection_name(&slug(INSTANCE))].points.len();
|
||||
assert_eq!(state.held_when_searched, [written], "the search saw every document");
|
||||
assert!(written >= pending as usize);
|
||||
}
|
||||
1415
crates/huginn-daemon/src/serve.rs
Normal file
1415
crates/huginn-daemon/src/serve.rs
Normal file
File diff suppressed because it is too large
Load diff
123
crates/huginn-daemon/tests/live_index_smoke.rs
Normal file
123
crates/huginn-daemon/tests/live_index_smoke.rs
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
//! A live check of the two adapters against a real Qdrant and a real Ollama.
|
||||
//! Ignored by default: it needs both services, and it writes to Qdrant, so it
|
||||
//! touches one scratch collection it creates (`huginn-smoke-<random>`, and a
|
||||
//! dotted sibling to learn whether Qdrant accepts a `.` in a name) and deletes
|
||||
//! both at the end. It never names, lists or reads any other collection.
|
||||
//!
|
||||
//! HUGINN_SMOKE_QDRANT=http://127.0.0.1:6333 \
|
||||
//! HUGINN_SMOKE_OLLAMA=http://10.77.0.4:11434 \
|
||||
//! HUGINN_SMOKE_MODEL=qwen3-embedding:0.6b \
|
||||
//! cargo test -p huginn-daemon --test live_index_smoke -- --ignored --nocapture
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::time::Duration;
|
||||
|
||||
use huginn_daemon::index::ollama::{OllamaEmbedder, QUERY_INSTRUCTION};
|
||||
use huginn_daemon::index::qdrant::QdrantIndex;
|
||||
use huginn_daemon::index::{
|
||||
CollectionMeta, Described, Embedder, MANAGED_BY, Point, PointPayload, VectorIndex, point_id,
|
||||
};
|
||||
use huginn_mind::INDEX_TEXT_VERSION;
|
||||
|
||||
fn env(name: &str) -> String {
|
||||
std::env::var(name).unwrap_or_else(|_| panic!("{name} is required"))
|
||||
}
|
||||
|
||||
fn raw(method: &str, url: &str, body: Option<&str>) -> (u16, String) {
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
let mut response = match (method, body) {
|
||||
("DELETE", _) => agent.delete(url).call(),
|
||||
("POST", Some(body)) => agent.post(url).header("content-type", "application/json").send(body),
|
||||
_ => agent.get(url).call(),
|
||||
}
|
||||
.expect("the request was sent");
|
||||
let status = response.status().as_u16();
|
||||
(status, response.body_mut().read_to_string().unwrap())
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore = "needs a live Qdrant and Ollama, and writes a scratch collection"]
|
||||
fn the_adapters_embed_write_query_and_clean_up_a_scratch_collection() {
|
||||
let (qdrant, ollama, model) = (env("HUGINN_SMOKE_QDRANT"), env("HUGINN_SMOKE_OLLAMA"), env("HUGINN_SMOKE_MODEL"));
|
||||
let stamp = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_nanos();
|
||||
let plain = format!("huginn-smoke-{:x}-{stamp:x}", std::process::id());
|
||||
let dotted = format!("{plain}.dotted");
|
||||
|
||||
let mut embedder = OllamaEmbedder::new(&ollama, &model);
|
||||
let identity = embedder.model_identity(Duration::from_secs(60)).expect("the model is listed");
|
||||
println!("identity: {identity:?}");
|
||||
let texts = ["The organ owns the mind.".to_string(), "A recipe for lentil soup.".to_string()];
|
||||
let vectors = embedder.embed(&texts, Duration::from_secs(60)).expect("two texts embed");
|
||||
assert_eq!(vectors.len(), 2);
|
||||
assert!(vectors.iter().all(|vector| vector.len() == identity.dimensions as usize));
|
||||
|
||||
let meta = CollectionMeta {
|
||||
managed_by: MANAGED_BY.into(),
|
||||
instance: "smoke".into(),
|
||||
mind: "mind-commit-smoke".into(),
|
||||
model: identity.name.clone(),
|
||||
model_digest: identity.digest.clone(),
|
||||
dimensions: identity.dimensions,
|
||||
index_text_version: INDEX_TEXT_VERSION,
|
||||
};
|
||||
let mut store = QdrantIndex::new(&qdrant);
|
||||
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
|
||||
for name in [&plain, &dotted] {
|
||||
assert_eq!(store.describe(name, Duration::from_secs(60)).unwrap(), Described::Absent);
|
||||
store.recreate(name, &meta).unwrap();
|
||||
assert_eq!(store.describe(name, Duration::from_secs(60)).unwrap(), Described::Labelled(meta.clone()));
|
||||
assert!(store.ids(name).unwrap().is_empty());
|
||||
let points: Vec<Point> = texts
|
||||
.iter()
|
||||
.zip(&vectors)
|
||||
.enumerate()
|
||||
.map(|(at, (text, vector))| Point {
|
||||
id: point_id(&format!("smoke:doc:{at}")),
|
||||
vector: vector.clone(),
|
||||
payload: PointPayload {
|
||||
doc_id: format!("smoke:doc:{at}"),
|
||||
kind: "question".into(),
|
||||
root: "smoke".into(),
|
||||
ordinal: at as u64 + 1,
|
||||
text_sha256: format!("{}", text.len()),
|
||||
},
|
||||
})
|
||||
.collect();
|
||||
store.upsert(name, &points).unwrap();
|
||||
let expected: BTreeSet<String> = points.iter().map(|point| point.id.clone()).collect();
|
||||
assert_eq!(store.ids(name).unwrap(), expected);
|
||||
// The nearest point to the first text is the first point.
|
||||
let query = serde_json_free_query(&vectors[0]);
|
||||
let (status, answer) = raw("POST", &format!("{qdrant}/collections/{name}/points/search"), Some(&query));
|
||||
assert_eq!(status, 200, "{answer}");
|
||||
assert!(answer.contains(&points[0].id), "{answer}");
|
||||
// The read side: a paraphrase that shares no exact words with the
|
||||
// first document finds it, through the adapter, with the query
|
||||
// instruction the worker uses.
|
||||
let asked = format!("Instruct: {QUERY_INSTRUCTION}
|
||||
Query: Who is responsible for keeping the memory?");
|
||||
let query = embedder.embed(&[asked], Duration::from_secs(60)).expect("the query embeds").remove(0);
|
||||
let hits = store.search(name, &query, 2, Duration::from_secs(60)).unwrap();
|
||||
assert_eq!(hits.len(), 2, "{hits:?}");
|
||||
assert_eq!((hits[0].doc_id.as_str(), hits[0].kind.as_str()), ("smoke:doc:0", "question"), "{hits:?}");
|
||||
assert!(hits[0].score > hits[1].score, "{hits:?}");
|
||||
println!("{name}: described, created, upserted {} points, queried, searched: {hits:?}", points.len());
|
||||
}
|
||||
}));
|
||||
for name in [&plain, &dotted] {
|
||||
let (status, _) = raw("DELETE", &format!("{qdrant}/collections/{name}?wait=true"), None);
|
||||
println!("deleted {name}: {status}");
|
||||
}
|
||||
let (status, _) = raw("GET", &format!("{qdrant}/collections/{plain}"), None);
|
||||
assert_eq!(status, 404, "the scratch collection is gone");
|
||||
if let Err(panic) = outcome {
|
||||
std::panic::resume_unwind(panic);
|
||||
}
|
||||
}
|
||||
|
||||
/// The nearest-neighbour body, written out by hand so the test needs no JSON
|
||||
/// crate of its own.
|
||||
fn serde_json_free_query(vector: &[f32]) -> String {
|
||||
let numbers: Vec<String> = vector.iter().map(|value| value.to_string()).collect();
|
||||
format!("{{\"vector\":[{}],\"limit\":1}}", numbers.join(","))
|
||||
}
|
||||
39
crates/huginn-mind/Cargo.toml
Normal file
39
crates/huginn-mind/Cargo.toml
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
[package]
|
||||
name = "huginn-mind"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
publish.workspace = true
|
||||
|
||||
# The organ's rule engine: storage, identity and admission of one instance's
|
||||
# mind. Document shape, bounds, formats and keys come from `epiphany-pipeline`;
|
||||
# the store comes from CultLib's Rust CultCache at the same rev Epiphany pins,
|
||||
# so the leaf's wrappers and this crate's cache are one `cultcache-rs`. The
|
||||
# read side selects through CultNet's typed selection (`cultnet-rs`, the same
|
||||
# rev the daemon pins): the selection's vocabulary and evaluator are the
|
||||
# substrate's, this crate supplies the rows. No socket, no index: those are the
|
||||
# daemon's (Cut 10, Cut 11).
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
base64 = "0.22"
|
||||
chrono = "0.4.44"
|
||||
cultcache-rs = { git = "https://github.com/GameCult/CultLib.git", rev = "8fc74c709444b884a2bd339e0c306bf9484d99d2", package = "cultcache-rs" }
|
||||
cultnet-rs = { git = "https://github.com/GameCult/CultLib.git", rev = "8fc74c709444b884a2bd339e0c306bf9484d99d2", package = "cultnet-rs" }
|
||||
epiphany-pipeline = { git = "https://github.com/GameCult/Epiphany.git", rev = "73edee5cdaddece8bda9c6c3271e8ddd064337f9", package = "epiphany-pipeline" }
|
||||
rmp-serde = "1"
|
||||
schemars = "1"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
sha2 = "0.10"
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json = "1"
|
||||
tempfile = "3"
|
||||
|
||||
# `test-support` exposes `Mind::store_path_for` and `Mind::open_with`, a
|
||||
# grammar-checked path accessor and the fail-closed open door, for tests in
|
||||
# other crates that need to reach a raw store directly. It buys no runtime
|
||||
# surface: gated `#[cfg(any(test, feature = "test-support"))]`, so it
|
||||
# compiles only under `cargo test` in this crate or when a dependent's own
|
||||
# dev-dependencies enable it (see huginn-daemon/Cargo.toml).
|
||||
[features]
|
||||
test-support = []
|
||||
2094
crates/huginn-mind/src/admission.rs
Normal file
2094
crates/huginn-mind/src/admission.rs
Normal file
File diff suppressed because it is too large
Load diff
443
crates/huginn-mind/src/docs.rs
Normal file
443
crates/huginn-mind/src/docs.rs
Normal file
|
|
@ -0,0 +1,443 @@
|
|||
//! The image and the batch, the two places a rule or a view may look, and
|
||||
//! every derivation over them. "In force" is recursive: a document is in force
|
||||
//! when no resolution that is itself in force names it, so a withdrawn
|
||||
//! resolution stops closing its subject. Computed over image and batch at rule
|
||||
//! time, over the image at read time, never stored, and never from
|
||||
//! `stored_at`.
|
||||
|
||||
use cultcache_rs::CultCacheEnvelope;
|
||||
use epiphany_pipeline::{
|
||||
OrgRepo, PipelineDocument, PipelineKind, PipelineRef, PipelineResolution, PipelineStewardship, ResolutionOutcome,
|
||||
Short, Slug,
|
||||
};
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::refusal::MindRefusal;
|
||||
|
||||
/// A batch document after A3 and A5: decoded, keyed, and its envelope.
|
||||
pub(crate) struct Staged {
|
||||
pub(crate) kind: PipelineKind,
|
||||
pub(crate) key: String,
|
||||
pub(crate) document: PipelineDocument,
|
||||
pub(crate) envelope: CultCacheEnvelope,
|
||||
}
|
||||
|
||||
/// An image document, decoded.
|
||||
pub(crate) struct Held {
|
||||
pub(crate) kind: PipelineKind,
|
||||
pub(crate) key: String,
|
||||
pub(crate) document: PipelineDocument,
|
||||
}
|
||||
|
||||
/// The image and the batch, the two places a rule may look.
|
||||
pub(crate) struct Docs {
|
||||
pub(crate) image: Vec<Held>,
|
||||
pub(crate) batch: Vec<Staged>,
|
||||
}
|
||||
|
||||
pub(crate) fn kind_of_type(type_id: &str) -> Option<PipelineKind> {
|
||||
PipelineKind::ALL.iter().copied().find(|kind| kind.type_id() == type_id)
|
||||
}
|
||||
|
||||
impl Docs {
|
||||
pub(crate) fn from_image(envelopes: &[CultCacheEnvelope]) -> Result<Self, MindRefusal> {
|
||||
let mut image = Vec::new();
|
||||
for envelope in envelopes {
|
||||
let Some(kind) = kind_of_type(&envelope.r#type) else { continue };
|
||||
let document = PipelineDocument::decode(envelope).map_err(|refusal| MindRefusal::Unavailable {
|
||||
detail: format!("stored document {} does not decode: {refusal}", envelope.key),
|
||||
})?;
|
||||
image.push(Held { kind, key: envelope.key.clone(), document });
|
||||
}
|
||||
Ok(Self { image, batch: Vec::new() })
|
||||
}
|
||||
|
||||
/// A4: one identity, once.
|
||||
pub(crate) fn push(&mut self, staged: Staged) -> Result<(), MindRefusal> {
|
||||
if self.batch.iter().any(|other| other.kind == staged.kind && other.key == staged.key) {
|
||||
return Err(MindRefusal::IdentityCollision { kind: staged.kind, id: staged.key });
|
||||
}
|
||||
self.batch.push(staged);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn in_batch(&self, kind: PipelineKind, id: &str) -> Option<&PipelineDocument> {
|
||||
self.batch.iter().find(|staged| staged.kind == kind && staged.key == id).map(|staged| &staged.document)
|
||||
}
|
||||
|
||||
pub(crate) fn in_image(&self, kind: PipelineKind, id: &str) -> Option<&PipelineDocument> {
|
||||
self.image.iter().find(|held| held.kind == kind && held.key == id).map(|held| &held.document)
|
||||
}
|
||||
|
||||
pub(crate) fn find(&self, kind: PipelineKind, id: &str) -> Option<&PipelineDocument> {
|
||||
self.in_batch(kind, id).or_else(|| self.in_image(kind, id))
|
||||
}
|
||||
|
||||
pub(crate) fn of_kind(&self, kind: PipelineKind) -> impl Iterator<Item = (&str, &PipelineDocument)> {
|
||||
let batch = self.batch.iter().filter(move |staged| staged.kind == kind).map(|staged| (staged.key.as_str(), &staged.document));
|
||||
let image = self.image.iter().filter(move |held| held.kind == kind).map(|held| (held.key.as_str(), &held.document));
|
||||
batch.chain(image)
|
||||
}
|
||||
|
||||
pub(crate) fn resolutions(&self) -> impl Iterator<Item = (&str, &PipelineResolution)> {
|
||||
self.of_kind(PipelineKind::Resolution).filter_map(|(key, document)| match document {
|
||||
PipelineDocument::Resolution(resolution) => Some((key, resolution)),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn stewardships(&self) -> impl Iterator<Item = (&str, &PipelineStewardship)> {
|
||||
self.of_kind(PipelineKind::Stewardship).filter_map(|(key, document)| match document {
|
||||
PipelineDocument::Stewardship(stewardship) => Some((key, stewardship)),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
/// Every resolution of one subject, in image and batch: the scope a
|
||||
/// subject's sequence counts in.
|
||||
pub(crate) fn resolutions_of(&self, subject: &PipelineRef) -> impl Iterator<Item = (&str, &PipelineResolution)> {
|
||||
self.resolutions().filter(move |(_, resolution)| resolution.subject == *subject)
|
||||
}
|
||||
|
||||
/// Every assignment of one repo to one mind, in image and batch, in force
|
||||
/// or not: the scope a stewardship's sequence counts in.
|
||||
pub(crate) fn assignments_of(&self, mind: &Slug, repo: &OrgRepo) -> impl Iterator<Item = (&str, &PipelineStewardship)> {
|
||||
self.stewardships().filter(move |(_, stewardship)| stewardship.instance == *mind && stewardship.repo == *repo)
|
||||
}
|
||||
|
||||
/// The greatest sequence among a subject's resolutions in image or batch,
|
||||
/// `0` when it has none, so the next is `latest + 1`. `own` is the
|
||||
/// document being checked, excluded by content identity so that an exact
|
||||
/// replay measures the same `latest` the first admission did.
|
||||
pub(crate) fn latest_resolution(&self, subject: &PipelineRef, own: Option<&PipelineResolution>) -> u32 {
|
||||
self.resolutions_of(subject)
|
||||
.filter(|(_, resolution)| own != Some(*resolution))
|
||||
.map(|(_, resolution)| resolution.sequence)
|
||||
.max()
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// The sequence a derivation takes: the one the record it already derived
|
||||
/// carries, when image or batch holds that record, so an exact replay
|
||||
/// re-derives the same document and A9 answers with the stored receipt;
|
||||
/// otherwise the next. A derivation is recognised by what it writes into
|
||||
/// its record: a resolution by its outcome, an assignment by its note.
|
||||
pub(crate) fn derived_resolution_sequence(
|
||||
&self,
|
||||
subject: &PipelineRef,
|
||||
derived: impl Fn(&ResolutionOutcome) -> bool,
|
||||
) -> u32 {
|
||||
self.resolutions()
|
||||
.find(|(_, resolution)| resolution.subject == *subject && derived(&resolution.outcome))
|
||||
.map_or_else(|| self.latest_resolution(subject, None) + 1, |(_, resolution)| resolution.sequence)
|
||||
}
|
||||
|
||||
/// The same for the assignment a hand-off derives, which notes its key.
|
||||
pub(crate) fn derived_stewardship_sequence(&self, mind: &Slug, repo: &OrgRepo, note: &str) -> u32 {
|
||||
self.stewardships()
|
||||
.find(|(_, stewardship)| stewardship.instance == *mind && stewardship.repo == *repo && stewardship.note.0 == note)
|
||||
.map_or_else(|| self.latest_stewardship(mind, repo, None) + 1, |(_, stewardship)| stewardship.sequence)
|
||||
}
|
||||
|
||||
/// The same for a repo's assignments to one mind.
|
||||
pub(crate) fn latest_stewardship(&self, mind: &Slug, repo: &OrgRepo, own: Option<&PipelineStewardship>) -> u32 {
|
||||
self.assignments_of(mind, repo)
|
||||
.filter(|(_, stewardship)| own != Some(*stewardship))
|
||||
.map(|(_, stewardship)| stewardship.sequence)
|
||||
.max()
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// No resolution that is itself in force names the document. The
|
||||
/// recursion is the whole rule: a withdrawn resolution stops counting, so
|
||||
/// its subject is open again. `key(R)` is strictly longer than the id it
|
||||
/// resolves, so the chain is bounded and this terminates.
|
||||
pub(crate) fn in_force(&self, kind: PipelineKind, id: &str) -> bool {
|
||||
self.in_force_unless(kind, id, |_| false)
|
||||
}
|
||||
|
||||
/// In force, ignoring a resolution the caller's own document is or
|
||||
/// derives: the ruling that answers a question is what resolves it, the
|
||||
/// hand-off that withdraws a stewardship is what resolves that, and a
|
||||
/// resolution does not resolve its own subject out from under itself.
|
||||
pub(crate) fn in_force_unless(&self, kind: PipelineKind, id: &str, own: impl Fn(&PipelineResolution) -> bool) -> bool {
|
||||
self.closing_resolution_unless(kind, id, own).is_none()
|
||||
}
|
||||
|
||||
/// The in-force resolution naming the document, other than `own`, if any;
|
||||
/// exact rather than a tie-break because A8 and the Q19 cap admit at most
|
||||
/// one.
|
||||
pub(crate) fn closing_resolution_unless(
|
||||
&self,
|
||||
kind: PipelineKind,
|
||||
id: &str,
|
||||
own: impl Fn(&PipelineResolution) -> bool,
|
||||
) -> Option<(&str, &PipelineResolution)> {
|
||||
self.resolutions().find(|&(key, resolution)| {
|
||||
resolution.subject.kind == kind
|
||||
&& resolution.subject.id.0 == id
|
||||
&& !own(resolution)
|
||||
&& self.in_force(PipelineKind::Resolution, key)
|
||||
})
|
||||
}
|
||||
|
||||
/// The resolution that closes the document, if any: the views' status,
|
||||
/// derived where admission derives whether there is one at all.
|
||||
pub(crate) fn closing_resolution(&self, kind: PipelineKind, id: &str) -> Option<(&str, &PipelineResolution)> {
|
||||
self.closing_resolution_unless(kind, id, |_| false)
|
||||
}
|
||||
|
||||
/// Every in-force stewardship of `(mind, repo)`, ignoring the withdrawal
|
||||
/// `hand_off_key` derives. The stewardship row holds one in force at a
|
||||
/// time, but that exclusion can put a second back in the list: the one a
|
||||
/// replayed hand-off withdrew, beside whatever stewards the repo now.
|
||||
pub(crate) fn stewardships_of(
|
||||
&self,
|
||||
mind: &Slug,
|
||||
repo: &OrgRepo,
|
||||
hand_off_key: Option<&str>,
|
||||
) -> Vec<(&str, &PipelineStewardship)> {
|
||||
self.assignments_of(mind, repo)
|
||||
.filter(|(key, _)| {
|
||||
self.in_force_unless(PipelineKind::Stewardship, key, |resolution| {
|
||||
matches!(&resolution.outcome, ResolutionOutcome::Withdrawn { reason } if Some(reason.0.as_str()) == hand_off_key)
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The in-force record that stands later than `base` in the scope `base`
|
||||
/// is sequenced in, if any: what a withdrawal that put `base` back in
|
||||
/// force would leave standing beside it.
|
||||
pub(crate) fn later_in_force(&self, base: &PipelineDocument) -> Option<&str> {
|
||||
let kind = base.kind();
|
||||
self.of_kind(kind).find(|(key, other)| later_than(base, other) && self.in_force(kind, key)).map(|(key, _)| key)
|
||||
}
|
||||
|
||||
/// The stewardship of `(mind, repo)` a hand-off acts on: the one whose
|
||||
/// withdrawal already carries `hand_off_key`, when image or batch holds
|
||||
/// that withdrawal, so a replay derives the same withdrawal again and A9
|
||||
/// answers with the stored receipt; otherwise the one in force. Key order
|
||||
/// cannot pick between the two the exclusion leaves standing -- once a
|
||||
/// repo has changed hands ten times, `n11` sorts before `n2` -- so the
|
||||
/// content match picks, as it does for the sequence the other derivations
|
||||
/// take.
|
||||
pub(crate) fn stewardship_of(
|
||||
&self,
|
||||
mind: &Slug,
|
||||
repo: &OrgRepo,
|
||||
hand_off_key: Option<&str>,
|
||||
) -> Option<(&str, &PipelineStewardship)> {
|
||||
let candidates = self.stewardships_of(mind, repo, hand_off_key);
|
||||
hand_off_key
|
||||
.and_then(|hand_off| {
|
||||
candidates.iter().find(|(key, _)| {
|
||||
self.resolutions().any(|(_, resolution)| {
|
||||
resolution.subject.kind == PipelineKind::Stewardship
|
||||
&& resolution.subject.id.0 == *key
|
||||
&& matches!(&resolution.outcome, ResolutionOutcome::Withdrawn { reason } if reason.0 == hand_off)
|
||||
})
|
||||
})
|
||||
})
|
||||
.or_else(|| candidates.first())
|
||||
.copied()
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `other` is a later record than `base` in the scope `base` is
|
||||
/// sequenced in: `(instance, repo)` for a stewardship, the campaign for a
|
||||
/// target and the cut for a cut spec. Kinds that carry no sequence have no
|
||||
/// scope and no order, and a resolution is never a base here: the cap above
|
||||
/// refuses the withdrawal that would reinstate one before the question of a
|
||||
/// later record arises.
|
||||
fn later_than(base: &PipelineDocument, other: &PipelineDocument) -> bool {
|
||||
use PipelineDocument as D;
|
||||
match (base, other) {
|
||||
(D::Stewardship(base), D::Stewardship(other)) => {
|
||||
other.instance == base.instance && other.repo == base.repo && other.sequence > base.sequence
|
||||
}
|
||||
(D::Target(base), D::Target(other)) => other.campaign == base.campaign && other.revision > base.revision,
|
||||
(D::CutSpec(base), D::CutSpec(other)) => {
|
||||
other.campaign == base.campaign && other.cut == base.cut && other.revision > base.revision
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// The referring field of a citation edge, one per leaf field that names
|
||||
/// another document. The wire spelling is the field's own name, so a client
|
||||
/// reads an edge's role as the field it came from.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, JsonSchema)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CitationRole {
|
||||
RaisedIn,
|
||||
Answers,
|
||||
Rulings,
|
||||
Questions,
|
||||
CutSpec,
|
||||
Forks,
|
||||
CutReport,
|
||||
Findings,
|
||||
Verdict,
|
||||
Source,
|
||||
Subject,
|
||||
SupersededBy,
|
||||
ResolvedBy,
|
||||
DeferredTo,
|
||||
Documents,
|
||||
}
|
||||
|
||||
impl CitationRole {
|
||||
pub const ALL: [CitationRole; 15] = [
|
||||
Self::RaisedIn,
|
||||
Self::Answers,
|
||||
Self::Rulings,
|
||||
Self::Questions,
|
||||
Self::CutSpec,
|
||||
Self::Forks,
|
||||
Self::CutReport,
|
||||
Self::Findings,
|
||||
Self::Verdict,
|
||||
Self::Source,
|
||||
Self::Subject,
|
||||
Self::SupersededBy,
|
||||
Self::ResolvedBy,
|
||||
Self::DeferredTo,
|
||||
Self::Documents,
|
||||
];
|
||||
|
||||
/// The wire spelling, equal to the referring field's name.
|
||||
pub fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::RaisedIn => "raised_in",
|
||||
Self::Answers => "answers",
|
||||
Self::Rulings => "rulings",
|
||||
Self::Questions => "questions",
|
||||
Self::CutSpec => "cut_spec",
|
||||
Self::Forks => "forks",
|
||||
Self::CutReport => "cut_report",
|
||||
Self::Findings => "findings",
|
||||
Self::Verdict => "verdict",
|
||||
Self::Source => "source",
|
||||
Self::Subject => "subject",
|
||||
Self::SupersededBy => "superseded_by",
|
||||
Self::ResolvedBy => "resolved_by",
|
||||
Self::DeferredTo => "deferred_to",
|
||||
Self::Documents => "documents",
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_name(name: &str) -> Option<Self> {
|
||||
Self::ALL.into_iter().find(|role| role.name() == name)
|
||||
}
|
||||
|
||||
/// The kind whose documents carry this field.
|
||||
pub fn carrier(self) -> PipelineKind {
|
||||
use PipelineKind as K;
|
||||
match self {
|
||||
Self::RaisedIn => K::Question,
|
||||
Self::Answers => K::Ruling,
|
||||
Self::Rulings | Self::Questions => K::CutSpec,
|
||||
Self::CutSpec | Self::Forks => K::CutReport,
|
||||
Self::CutReport | Self::Findings => K::Verdict,
|
||||
Self::Verdict => K::Finding,
|
||||
Self::Source => K::FollowUp,
|
||||
Self::Subject | Self::SupersededBy | Self::ResolvedBy | Self::DeferredTo => K::Resolution,
|
||||
Self::Documents => K::HandOff,
|
||||
}
|
||||
}
|
||||
|
||||
/// The kinds a referent of this field may be: the one kind the field is
|
||||
/// typed to, or every kind for the fields typed `PipelineRef` or a full
|
||||
/// id of any kind. "Any" is spelled as all thirteen, never as none.
|
||||
pub fn target_kinds(self) -> Vec<PipelineKind> {
|
||||
use PipelineKind as K;
|
||||
match self {
|
||||
Self::Answers | Self::Questions | Self::Forks => vec![K::Question],
|
||||
Self::Rulings => vec![K::Ruling],
|
||||
Self::CutSpec => vec![K::CutSpec],
|
||||
Self::CutReport => vec![K::CutReport],
|
||||
Self::Findings => vec![K::Finding],
|
||||
Self::Verdict => vec![K::Verdict],
|
||||
Self::RaisedIn
|
||||
| Self::Source
|
||||
| Self::Subject
|
||||
| Self::SupersededBy
|
||||
| Self::ResolvedBy
|
||||
| Self::DeferredTo
|
||||
| Self::Documents => PipelineKind::ALL.to_vec(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The kind segment of a full id, for the fields typed `Short` whose referent
|
||||
/// may be of any kind.
|
||||
pub(crate) fn kind_of_id(id: &str) -> Option<PipelineKind> {
|
||||
let name = id.split(':').nth(1)?;
|
||||
PipelineKind::ALL.iter().copied().find(|kind| kind.name() == name)
|
||||
}
|
||||
|
||||
fn cited(kind: PipelineKind, id: &Short) -> PipelineRef {
|
||||
PipelineRef { kind, id: id.clone() }
|
||||
}
|
||||
|
||||
/// What a resolution's outcome names, by the field that names it.
|
||||
pub(crate) fn outcome_citations(outcome: &ResolutionOutcome) -> Vec<(CitationRole, PipelineRef)> {
|
||||
match outcome {
|
||||
ResolutionOutcome::Superseded { by } => by.iter().map(|by| (CitationRole::SupersededBy, by.clone())).collect(),
|
||||
ResolutionOutcome::Answered { by } => vec![(CitationRole::ResolvedBy, by.clone())],
|
||||
ResolutionOutcome::Fixed { by: Some(by), .. } => vec![(CitationRole::ResolvedBy, by.clone())],
|
||||
ResolutionOutcome::Deferred { to } => vec![(CitationRole::DeferredTo, to.clone())],
|
||||
ResolutionOutcome::Fixed { by: None, .. } | ResolutionOutcome::Recorded { .. } | ResolutionOutcome::Withdrawn { .. } => {
|
||||
vec![]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Every citation edge a document carries, derived from its leaf fields each
|
||||
/// time it is asked and never stored: the one edge list. Admission's A7 reads
|
||||
/// it to require the referents, and the read side's rows read it as their
|
||||
/// references. `Fixed.commit`, `ForeignRef` precedents and the labels
|
||||
/// (`depends_on`, `invariants`, `promise`) are not edges. A hand-off's
|
||||
/// `documents` entry whose kind segment does not read cites nothing here;
|
||||
/// admission refuses such an entry on the source side before it asks.
|
||||
pub(crate) fn citations(document: &PipelineDocument) -> Vec<(CitationRole, PipelineRef)> {
|
||||
use CitationRole as R;
|
||||
use PipelineDocument as D;
|
||||
use PipelineKind as K;
|
||||
let mut edges = Vec::new();
|
||||
match document {
|
||||
D::Question(question) => edges.extend(question.raised_in.iter().map(|to| (R::RaisedIn, to.clone()))),
|
||||
D::Ruling(ruling) => edges.extend(ruling.answers.iter().map(|to| (R::Answers, cited(K::Question, to)))),
|
||||
D::CutSpec(spec) => {
|
||||
edges.extend(spec.rulings.iter().map(|to| (R::Rulings, cited(K::Ruling, to))));
|
||||
edges.extend(spec.questions.iter().map(|to| (R::Questions, cited(K::Question, to))));
|
||||
}
|
||||
D::CutReport(report) => {
|
||||
edges.push((R::CutSpec, cited(K::CutSpec, &report.cut_spec)));
|
||||
edges.extend(report.forks.iter().map(|to| (R::Forks, cited(K::Question, to))));
|
||||
}
|
||||
D::Verdict(verdict) => {
|
||||
edges.push((R::CutReport, cited(K::CutReport, &verdict.cut_report)));
|
||||
edges.extend(
|
||||
verdict
|
||||
.claims
|
||||
.iter()
|
||||
.flat_map(|claim| claim.findings.iter())
|
||||
.map(|to| (R::Findings, cited(K::Finding, to))),
|
||||
);
|
||||
}
|
||||
D::Finding(finding) => edges.push((R::Verdict, cited(K::Verdict, &finding.verdict))),
|
||||
D::FollowUp(follow_up) => edges.push((R::Source, follow_up.source.clone())),
|
||||
D::Resolution(resolution) => {
|
||||
edges.push((R::Subject, resolution.subject.clone()));
|
||||
edges.extend(outcome_citations(&resolution.outcome));
|
||||
}
|
||||
D::HandOff(hand_off) => edges.extend(
|
||||
hand_off.documents.iter().filter_map(|to| kind_of_id(&to.0).map(|kind| (R::Documents, cited(kind, to)))),
|
||||
),
|
||||
D::Campaign(_) | D::Target(_) | D::Instance(_) | D::Stewardship(_) => {}
|
||||
}
|
||||
edges
|
||||
}
|
||||
228
crates/huginn-mind/src/envelope.rs
Normal file
228
crates/huginn-mind/src/envelope.rs
Normal file
|
|
@ -0,0 +1,228 @@
|
|||
//! CultNet's operation envelope around the mind's vocabulary. Pure functions:
|
||||
//! a request goes out as `cultnet.operation_request.v0` and an answer comes
|
||||
//! back as `cultnet.operation_response.v0`, both carrying base64 of named
|
||||
//! MessagePack, which is what the C# reference and Eve's plugin ABI speak.
|
||||
//!
|
||||
//! A message this module cannot decode is a typed failure with a code, never
|
||||
//! a transport error and never a call into a mind. `payload_encoding` is
|
||||
//! CultNet's own check in `validate_message` and is not re-checked here.
|
||||
//! `source_runtime_id` and `target_runtime_id` on a request are read by
|
||||
//! nothing: ruling 18 says an identity is declared in the payload, not
|
||||
//! granted by the envelope.
|
||||
|
||||
use anyhow::Result;
|
||||
use base64::Engine;
|
||||
use base64::engine::general_purpose::STANDARD;
|
||||
use cultnet_rs::CultNetMessage;
|
||||
use crate::wire::{HuginnMindRequest, HuginnMindResponse, MIND_REQUEST_SCHEMA, MIND_RESPONSE_SCHEMA, MIND_SERVICE_ID};
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// The C# reference's `CultNetOperationServer.FailureSchemaId`. The Rust
|
||||
/// `cultnet-rs` publishes no type for it, so this two-field mirror carries the
|
||||
/// reference's keys; parity belongs in CultLib.
|
||||
pub const FAILURE_SCHEMA: &str = "gamecult.cultnet.operation_failure.v1";
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct OperationFailure {
|
||||
pub code: String,
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
impl OperationFailure {
|
||||
fn new(code: &str, message: impl Into<String>) -> Self {
|
||||
Self { code: code.into(), message: message.into() }
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_payload<T: Serialize>(value: &T) -> Result<String> {
|
||||
Ok(STANDARD.encode(rmp_serde::to_vec_named(value)?))
|
||||
}
|
||||
|
||||
/// One request, addressed to the mind service and naming its own operation.
|
||||
pub fn encode_request(
|
||||
message_id: &str,
|
||||
request: &HuginnMindRequest,
|
||||
source_runtime_id: Option<String>,
|
||||
) -> Result<CultNetMessage> {
|
||||
Ok(CultNetMessage::OperationRequest {
|
||||
message_id: message_id.to_string(),
|
||||
service_id: MIND_SERVICE_ID.to_string(),
|
||||
operation: request.operation().to_string(),
|
||||
payload_schema: MIND_REQUEST_SCHEMA.to_string(),
|
||||
payload_encoding: "messagepack-base64".to_string(),
|
||||
payload: encode_payload(request)?,
|
||||
source_runtime_id,
|
||||
target_runtime_id: None,
|
||||
})
|
||||
}
|
||||
|
||||
/// The message, or the failure that says why it is not a request of this
|
||||
/// service. The envelope's `operation` string is checked against the decoded
|
||||
/// request rather than trusted, so a payload cannot arrive under another
|
||||
/// operation's name.
|
||||
pub fn decode_request(message: &CultNetMessage) -> Result<(String, HuginnMindRequest), OperationFailure> {
|
||||
let CultNetMessage::OperationRequest { message_id, service_id, operation, payload_schema, payload, .. } = message
|
||||
else {
|
||||
return Err(OperationFailure::new(
|
||||
"not-an-operation-request",
|
||||
"huginn.mind reads cultnet.operation_request.v0",
|
||||
));
|
||||
};
|
||||
if service_id != MIND_SERVICE_ID {
|
||||
return Err(OperationFailure::new("wrong-service", format!("{service_id} is not {MIND_SERVICE_ID}")));
|
||||
}
|
||||
if payload_schema != MIND_REQUEST_SCHEMA {
|
||||
return Err(OperationFailure::new(
|
||||
"wrong-payload-schema",
|
||||
format!("{payload_schema} is not {MIND_REQUEST_SCHEMA}"),
|
||||
));
|
||||
}
|
||||
let bytes = STANDARD
|
||||
.decode(payload)
|
||||
.map_err(|error| OperationFailure::new("payload-not-base64", error.to_string()))?;
|
||||
let request: HuginnMindRequest = rmp_serde::from_slice(&bytes)
|
||||
.map_err(|error| OperationFailure::new("payload-not-a-request", error.to_string()))?;
|
||||
if operation != request.operation() {
|
||||
return Err(OperationFailure::new(
|
||||
"operation-mismatch",
|
||||
format!("the envelope says {operation} and the payload is {}", request.operation()),
|
||||
));
|
||||
}
|
||||
Ok((message_id.clone(), request))
|
||||
}
|
||||
|
||||
/// One answer, whose status the response derives.
|
||||
pub fn encode_response(
|
||||
message_id: &str,
|
||||
operation: &str,
|
||||
response: &HuginnMindResponse,
|
||||
source_runtime_id: &str,
|
||||
) -> Result<CultNetMessage> {
|
||||
Ok(CultNetMessage::OperationResponse {
|
||||
message_id: message_id.to_string(),
|
||||
service_id: MIND_SERVICE_ID.to_string(),
|
||||
operation: operation.to_string(),
|
||||
status: response.status().to_string(),
|
||||
payload_schema: MIND_RESPONSE_SCHEMA.to_string(),
|
||||
payload_encoding: "messagepack-base64".to_string(),
|
||||
payload: encode_payload(response)?,
|
||||
diagnostics: vec![],
|
||||
source_runtime_id: Some(source_runtime_id.to_string()),
|
||||
})
|
||||
}
|
||||
|
||||
/// The answer to an envelope that never reached a mind. Infallible: the
|
||||
/// failure is two strings and the daemon must always be able to say no.
|
||||
pub fn encode_failure(
|
||||
message_id: &str,
|
||||
operation: &str,
|
||||
failure: &OperationFailure,
|
||||
source_runtime_id: &str,
|
||||
) -> CultNetMessage {
|
||||
CultNetMessage::OperationResponse {
|
||||
message_id: message_id.to_string(),
|
||||
service_id: MIND_SERVICE_ID.to_string(),
|
||||
operation: operation.to_string(),
|
||||
status: "rejected".to_string(),
|
||||
payload_schema: FAILURE_SCHEMA.to_string(),
|
||||
payload_encoding: "messagepack-base64".to_string(),
|
||||
payload: STANDARD.encode(rmp_serde::to_vec_named(failure).expect("two strings encode")),
|
||||
diagnostics: vec![failure.code.clone()],
|
||||
source_runtime_id: Some(source_runtime_id.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// The client's side: the correlation key and either the mind's answer or the
|
||||
/// failure that says the envelope never reached one.
|
||||
pub fn decode_response(message: &CultNetMessage) -> Result<(String, Result<HuginnMindResponse, OperationFailure>)> {
|
||||
let CultNetMessage::OperationResponse { message_id, payload_schema, payload, .. } = message else {
|
||||
anyhow::bail!("huginn.mind answers with cultnet.operation_response.v0");
|
||||
};
|
||||
let bytes = STANDARD.decode(payload)?;
|
||||
let answer = match payload_schema.as_str() {
|
||||
MIND_RESPONSE_SCHEMA => Ok(rmp_serde::from_slice::<HuginnMindResponse>(&bytes)?),
|
||||
FAILURE_SCHEMA => Err(rmp_serde::from_slice::<OperationFailure>(&bytes)?),
|
||||
other => anyhow::bail!("{other} is neither {MIND_RESPONSE_SCHEMA} nor {FAILURE_SCHEMA}"),
|
||||
};
|
||||
Ok((message_id.clone(), answer))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::fixtures::{INSTANCE, OTHER_INSTANCE as OTHER, instance, provenance, slug};
|
||||
use crate::wire::MindStatus;
|
||||
use crate::{Faculty, MindRefusal, PipelineAdmissionBatch, PipelineAdmissionOutcome};
|
||||
|
||||
fn refusal() -> MindRefusal {
|
||||
MindRefusal::ForeignInstance { declared: OTHER.into(), mind: INSTANCE.into() }
|
||||
}
|
||||
|
||||
/// A refusal is an answer: it rides the response schema with a `rejected`
|
||||
/// status and decodes back to the same value. A conflict and a replay are
|
||||
/// answers about the mind, not rejections of the request.
|
||||
#[test]
|
||||
fn a_refusal_is_a_typed_response_not_a_transport_failure() {
|
||||
let cases = [
|
||||
(HuginnMindResponse::Refused(refusal()), "rejected"),
|
||||
(HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(refusal())), "rejected"),
|
||||
(
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Committed {
|
||||
receipt_id: "r1".into(),
|
||||
committed_at: "2026-09-16T12:00:00Z".into(),
|
||||
writes: vec![],
|
||||
}),
|
||||
"accepted",
|
||||
),
|
||||
(HuginnMindResponse::Admit(PipelineAdmissionOutcome::AlreadyAdmitted { receipt_id: "r1".into() }), "accepted"),
|
||||
(HuginnMindResponse::Admit(PipelineAdmissionOutcome::Conflict { identities: vec![] }), "accepted"),
|
||||
(
|
||||
HuginnMindResponse::Whoami(MindStatus {
|
||||
instance: slug(INSTANCE),
|
||||
schema_epoch: "epiphany.pipeline.epoch.v2".into(),
|
||||
documents: 0,
|
||||
receipts: 0,
|
||||
index: crate::wire::IndexStatus::Current,
|
||||
}),
|
||||
"accepted",
|
||||
),
|
||||
];
|
||||
for (response, expected) in cases {
|
||||
let message = encode_response("m-1", "admit", &response, "huginn-yggdrasil").unwrap();
|
||||
let CultNetMessage::OperationResponse { status, payload_schema, diagnostics, .. } = &message else {
|
||||
panic!("not an operation response");
|
||||
};
|
||||
assert_eq!(status, expected, "{response:?}");
|
||||
assert_eq!(payload_schema, MIND_RESPONSE_SCHEMA);
|
||||
assert!(diagnostics.is_empty());
|
||||
assert_eq!(decode_response(&message).unwrap(), ("m-1".into(), Ok(response)));
|
||||
}
|
||||
|
||||
let failure = OperationFailure::new("wrong-service", "odin.catalog is not huginn.mind");
|
||||
let message = encode_failure("m-2", "query", &failure, "huginn-yggdrasil");
|
||||
let CultNetMessage::OperationResponse { status, payload_schema, diagnostics, .. } = &message else {
|
||||
panic!("not an operation response");
|
||||
};
|
||||
assert_eq!((status.as_str(), payload_schema.as_str()), ("rejected", FAILURE_SCHEMA));
|
||||
assert_eq!(diagnostics, &vec!["wrong-service".to_string()]);
|
||||
assert_eq!(decode_response(&message).unwrap(), ("m-2".into(), Err(failure)));
|
||||
}
|
||||
|
||||
/// The round trip the client and the daemon share, and the one code that
|
||||
/// `serve::answer` cannot reach because it answers a foreign message with
|
||||
/// `Error` instead.
|
||||
#[test]
|
||||
fn a_request_round_trips_and_a_foreign_message_is_not_a_request() {
|
||||
let request = HuginnMindRequest::Admit(PipelineAdmissionBatch {
|
||||
instance: slug(INSTANCE),
|
||||
provenance: provenance(Faculty::Hands),
|
||||
documents: vec![instance(INSTANCE)],
|
||||
});
|
||||
let message = encode_request("m-3", &request, Some("eureka-state".into())).unwrap();
|
||||
assert_eq!(decode_request(&message).unwrap(), ("m-3".into(), request));
|
||||
|
||||
let foreign = CultNetMessage::Error { error: "not for you".into(), code: None, details: None };
|
||||
assert_eq!(decode_request(&foreign).unwrap_err().code, "not-an-operation-request");
|
||||
}
|
||||
}
|
||||
412
crates/huginn-mind/src/fixtures.rs
Normal file
412
crates/huginn-mind/src/fixtures.rs
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
//! Test fixtures shared by the modules' tests: one valid document of each
|
||||
//! kind, prepared through the leaf against the organ's cache, and a seeded
|
||||
//! mind to admit into.
|
||||
|
||||
use chrono::{DateTime, TimeZone, Utc};
|
||||
use cultcache_rs::CultCacheEnvelope;
|
||||
use epiphany_pipeline::{
|
||||
AuthorityMap, ClaimOutcome, CodeLocation, CommitRange, CutVerification, Date, DocRef, Evidence, EvidenceKind,
|
||||
FindingConfidence, FindingOrigin, FindingSeverity, Label, Line, MutationRecord, OrgRepo, PipelineCampaign,
|
||||
PipelineCutReport, PipelineCutSpec, PipelineDocument, PipelineFinding, PipelineFollowUp, PipelineHandOff,
|
||||
PipelineInstance, PipelineKind, PipelineQuestion, PipelineRef, PipelineResolution, PipelineRuling,
|
||||
PipelineStewardship, PipelineTarget, PipelineVerdict, Promise, QuestionOption, ReportCommit, ResolutionOutcome,
|
||||
RulingAuthority, Sha, Short, Slug, StructuralDelta, TargetInvariant, VerdictClaim,
|
||||
};
|
||||
|
||||
use crate::admission::{PipelineAdmissionBatch, PipelineAdmissionOutcome};
|
||||
use crate::mind::{HuginnMindEpoch, Mind, schema_cache};
|
||||
use crate::receipt::{Faculty, PipelineProvenance};
|
||||
use crate::refusal::MindRefusal;
|
||||
use crate::store::MindStore;
|
||||
use crate::store::test_stores::MemoryStore;
|
||||
|
||||
pub(crate) const INSTANCE: &str = "yggdrasil";
|
||||
pub(crate) const OTHER_INSTANCE: &str = "thought-cage";
|
||||
/// A foreign instance of `INSTANCE`'s own length, differing in its last byte
|
||||
/// alone: a comparison of lengths, of first bytes, or one gated on the
|
||||
/// declared name being the longer admits it, and a comparison of values does
|
||||
/// not.
|
||||
pub(crate) const NEAR_INSTANCE: &str = "yggdrasix";
|
||||
/// A foreign instance carrying `INSTANCE` whole as its prefix, so a prefix
|
||||
/// test admits it.
|
||||
pub(crate) const PREFIXED_INSTANCE: &str = "yggdrasil-two";
|
||||
/// A foreign instance differing from `INSTANCE` in case alone. A `Slug` is
|
||||
/// dot-joined labels, each `[A-Za-z0-9_-]{1,64}`, bounded to 64 bytes whole
|
||||
/// (`epiphany_pipeline::dotted_text`), so this is a name a client may declare
|
||||
/// and a mind may hold: two minds whose names differ only in case are two
|
||||
/// minds, and a comparison that folds case answers one of them for the other.
|
||||
pub(crate) const CASED_INSTANCE: &str = "Yggdrasil";
|
||||
/// Two foreign instances differing from `OTHER_INSTANCE` (`thought-cage`) in
|
||||
/// their separator alone: `_` and `.` are both legal in a `Slug` beside `-`
|
||||
/// (`_` inside a label, `.` joining labels), so `thought-cage`, `thought_cage`
|
||||
/// and `thought.cage` are three names, not one, and a comparison that folds a
|
||||
/// separator into another answers one of them for another.
|
||||
pub(crate) const UNDERSCORE_INSTANCE: &str = "thought_cage";
|
||||
pub(crate) const DOTTED_INSTANCE: &str = "thought.cage";
|
||||
pub(crate) const CAMPAIGN: &str = "eureka-state";
|
||||
pub(crate) const REPO: &str = "GameCult/Epiphany";
|
||||
pub(crate) const OTHER_REPO: &str = "GameCult/Huginn";
|
||||
pub(crate) const INVARIANT: &str = "mind-admits";
|
||||
|
||||
pub(crate) fn slug(value: &str) -> Slug {
|
||||
value.into()
|
||||
}
|
||||
|
||||
pub(crate) fn s(value: &str) -> Short {
|
||||
value.into()
|
||||
}
|
||||
|
||||
pub(crate) fn l(value: &str) -> Label {
|
||||
value.into()
|
||||
}
|
||||
|
||||
pub(crate) fn repo(value: &str) -> OrgRepo {
|
||||
value.into()
|
||||
}
|
||||
|
||||
pub(crate) fn date() -> Date {
|
||||
Date("2026-09-16".into())
|
||||
}
|
||||
|
||||
pub(crate) fn sha() -> Sha {
|
||||
Sha("5f98228d".into())
|
||||
}
|
||||
|
||||
pub(crate) fn id(kind: &str, local: &str) -> String {
|
||||
format!("{CAMPAIGN}:{kind}:{local}")
|
||||
}
|
||||
|
||||
pub(crate) fn r(kind: PipelineKind, id: &str) -> PipelineRef {
|
||||
PipelineRef { kind, id: s(id) }
|
||||
}
|
||||
|
||||
fn range() -> CommitRange {
|
||||
CommitRange { base: sha(), head: sha() }
|
||||
}
|
||||
|
||||
fn location() -> CodeLocation {
|
||||
CodeLocation { path: s("crates/huginn-mind/src/admission.rs"), line: 1, end_line: None }
|
||||
}
|
||||
|
||||
pub(crate) fn evidence() -> Evidence {
|
||||
Evidence { kind: EvidenceKind::Test, locator: "cargo test".into(), result: "ok".into() }
|
||||
}
|
||||
|
||||
fn doc_ref() -> DocRef {
|
||||
DocRef { path: s("notes/eureka-pipeline-state-target.md"), start_line: 1, end_line: 9, commit: sha() }
|
||||
}
|
||||
|
||||
fn delta() -> StructuralDelta {
|
||||
StructuralDelta {
|
||||
lines_added: 0,
|
||||
lines_removed: 0,
|
||||
dependencies_added: vec![],
|
||||
dependencies_removed: vec![],
|
||||
formats_added: vec![],
|
||||
formats_removed: vec![],
|
||||
targets_added: vec![],
|
||||
targets_removed: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn instance(name: &str) -> PipelineDocument {
|
||||
PipelineDocument::Instance(PipelineInstance {
|
||||
instance: slug(name),
|
||||
display_name: s(&format!("{name} mind")),
|
||||
created_at: date(),
|
||||
host: s(name),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn stewardship(instance: &str, repo_name: &str) -> PipelineDocument {
|
||||
stewardship_n(instance, repo_name, 1)
|
||||
}
|
||||
|
||||
/// A repo's `n`th assignment to a mind. The sequence is per `(instance,
|
||||
/// repo)` and the leaf keys it last, so `n2` is the record of a repo
|
||||
/// transferred away and taken back.
|
||||
pub(crate) fn stewardship_n(instance: &str, repo_name: &str, sequence: u32) -> PipelineDocument {
|
||||
PipelineDocument::Stewardship(PipelineStewardship {
|
||||
instance: slug(instance),
|
||||
repo: repo(repo_name),
|
||||
sequence,
|
||||
assigned_on: date(),
|
||||
note: "assigned".into(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn campaign(repos: &[&str]) -> PipelineDocument {
|
||||
PipelineDocument::Campaign(PipelineCampaign {
|
||||
slug: slug(CAMPAIGN),
|
||||
title: "Eureka pipeline state".into(),
|
||||
repos: repos.iter().map(|name| repo(name)).collect(),
|
||||
working_branch: s("codex/eureka-pipeline-state"),
|
||||
target_doc: doc_ref(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn target(revision: u32, invariants: &[&str]) -> PipelineDocument {
|
||||
PipelineDocument::Target(PipelineTarget {
|
||||
campaign: slug(CAMPAIGN),
|
||||
revision,
|
||||
invariants: invariants
|
||||
.iter()
|
||||
.map(|label| TargetInvariant { label: l(label), statement: "Only admission writes.".into() })
|
||||
.collect(),
|
||||
not_in_scope: vec![],
|
||||
canonical_implementations: vec![],
|
||||
doc: doc_ref(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn question(label: &str, options: &[&str], recommended: &str) -> PipelineDocument {
|
||||
PipelineDocument::Question(PipelineQuestion {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: l(label),
|
||||
title: "Who owns the state?".into(),
|
||||
question: "Who owns the state?".into(),
|
||||
options: options.iter().map(|option| QuestionOption { label: l(option), text: "an option".into() }).collect(),
|
||||
recommended: l(recommended),
|
||||
depends: vec![],
|
||||
raised_in: None,
|
||||
asked_on: date(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn ruling(label: &str) -> PipelineRuling {
|
||||
PipelineRuling {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: l(label),
|
||||
title: "An instance owns its mind".into(),
|
||||
answers: None,
|
||||
choice: None,
|
||||
ruling: "An instance owns its mind.".into(),
|
||||
operator_quote: None,
|
||||
ruled_on: date(),
|
||||
precedents: vec![],
|
||||
authority: RulingAuthority::Standing,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn cut_spec(cut: &str, revision: u32) -> PipelineCutSpec {
|
||||
PipelineCutSpec {
|
||||
campaign: slug(CAMPAIGN),
|
||||
cut: l(cut),
|
||||
revision,
|
||||
title: "A cut".into(),
|
||||
repo: repo(REPO),
|
||||
branch: s("codex/eureka-pipeline-state"),
|
||||
base: sha(),
|
||||
depends_on: vec![],
|
||||
first: vec![],
|
||||
deletes: vec![],
|
||||
keeps_moves: vec![],
|
||||
adds: vec![],
|
||||
file_changes: vec![],
|
||||
authority_map: None::<AuthorityMap>,
|
||||
verification: CutVerification { builds: vec![], tests: vec![], negative: vec![], operator: vec![] },
|
||||
estimate: delta(),
|
||||
rulings: vec![],
|
||||
questions: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn cut_report(cut: &str, attempt: u32) -> PipelineCutReport {
|
||||
PipelineCutReport {
|
||||
campaign: slug(CAMPAIGN),
|
||||
cut_spec: s(&id("cut_spec", &format!("cut-{cut}.r1"))),
|
||||
attempt,
|
||||
repo: repo(REPO),
|
||||
branch: s("codex/eureka-pipeline-state"),
|
||||
commits: vec![ReportCommit { sha: sha(), subject: "Land the cut".into(), builds: true }],
|
||||
range: range(),
|
||||
verification: vec![evidence()],
|
||||
mutations: vec![MutationRecord {
|
||||
label: l("M1"),
|
||||
rule: "the rule".into(),
|
||||
location: location(),
|
||||
before: "before".into(),
|
||||
after: "after".into(),
|
||||
commit: sha(),
|
||||
failed_as_expected: true,
|
||||
}],
|
||||
deviations: vec![],
|
||||
forks: vec![],
|
||||
structural_delta: delta(),
|
||||
landed_names: vec![],
|
||||
undone: vec![],
|
||||
promises: vec![Promise { label: l("P1"), text: "One derived key per document.".into() }],
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn claim(outcome: ClaimOutcome, findings: &[&str], promise: Option<&str>, mutations: &[&str]) -> VerdictClaim {
|
||||
VerdictClaim {
|
||||
claim: Line(format!("{outcome:?} claim")),
|
||||
outcome,
|
||||
evidence: vec![evidence()],
|
||||
findings: findings.iter().map(|finding| s(finding)).collect(),
|
||||
promise: promise.map(l),
|
||||
mutations: mutations.iter().map(|label| l(label)).collect(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn verdict(cut: &str, pass: u32, claims: Vec<VerdictClaim>) -> PipelineDocument {
|
||||
PipelineDocument::Verdict(PipelineVerdict {
|
||||
campaign: slug(CAMPAIGN),
|
||||
cut_report: s(&id("cut_report", &format!("cut-{cut}.h1"))),
|
||||
pass,
|
||||
range: range(),
|
||||
claims,
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn finding(cut: &str, pass: u32, label: &str, confidence: FindingConfidence) -> PipelineFinding {
|
||||
PipelineFinding {
|
||||
campaign: slug(CAMPAIGN),
|
||||
verdict: s(&id("verdict", &format!("cut-{cut}.s{pass}"))),
|
||||
label: l(label),
|
||||
range: range(),
|
||||
confidence,
|
||||
severity: FindingSeverity::High,
|
||||
claim: "A dotted label composes two keys.".into(),
|
||||
invariants: vec![l(INVARIANT)],
|
||||
locations: vec![location()],
|
||||
failure_scenario: "Two documents claim one key.".into(),
|
||||
evidence: vec![evidence()],
|
||||
precedents: vec![],
|
||||
origin: FindingOrigin::Introduced,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn follow_up(label: &str, source: PipelineRef) -> PipelineDocument {
|
||||
PipelineDocument::FollowUp(PipelineFollowUp {
|
||||
campaign: slug(CAMPAIGN),
|
||||
label: l(label),
|
||||
source,
|
||||
repo: repo(REPO),
|
||||
locations: vec![location()],
|
||||
item: "Per-kind admission rules.".into(),
|
||||
why_it_can_wait: "The organ owns admission.".into(),
|
||||
owner: s("Hands"),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn resolution(subject: PipelineRef, outcome: ResolutionOutcome) -> PipelineDocument {
|
||||
resolution_n(subject, 1, outcome)
|
||||
}
|
||||
|
||||
/// A subject's `n`th resolution. The sequence is per subject and the leaf
|
||||
/// keys it last, so a withdrawn closure stays under its subject and the next
|
||||
/// one is nameable.
|
||||
pub(crate) fn resolution_n(subject: PipelineRef, sequence: u32, outcome: ResolutionOutcome) -> PipelineDocument {
|
||||
PipelineDocument::Resolution(PipelineResolution {
|
||||
subject,
|
||||
sequence,
|
||||
outcome,
|
||||
rationale: "Resolved.".into(),
|
||||
resolved_on: date(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn withdrawn() -> ResolutionOutcome {
|
||||
ResolutionOutcome::Withdrawn { reason: "moot".into() }
|
||||
}
|
||||
|
||||
pub(crate) fn superseded(by: &[PipelineRef]) -> ResolutionOutcome {
|
||||
ResolutionOutcome::Superseded { by: by.to_vec() }
|
||||
}
|
||||
|
||||
pub(crate) fn hand_off(from: &str, to: &str, repo_name: &str, documents: &[&str]) -> PipelineDocument {
|
||||
PipelineDocument::HandOff(PipelineHandOff {
|
||||
from_instance: slug(from),
|
||||
to_instance: slug(to),
|
||||
repo: repo(repo_name),
|
||||
documents: documents.iter().map(|document| s(document)).collect(),
|
||||
reason: "The workstation mind takes the campaign.".into(),
|
||||
handed_on: date(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn provenance(faculty: Faculty) -> PipelineProvenance {
|
||||
PipelineProvenance { faculty, agent: s("claude"), session: s("session-1"), tool: s("admit") }
|
||||
}
|
||||
|
||||
pub(crate) fn now() -> DateTime<Utc> {
|
||||
Utc.with_ymd_and_hms(2026, 9, 16, 12, 0, 0).unwrap()
|
||||
}
|
||||
|
||||
/// The leaf's envelope for a document, keyed and encoded as the organ stores
|
||||
/// it.
|
||||
pub(crate) fn prepare(document: &PipelineDocument) -> CultCacheEnvelope {
|
||||
document.validate().unwrap();
|
||||
document.prepare(&schema_cache().unwrap()).unwrap()
|
||||
}
|
||||
|
||||
/// The epoch record admission derives on the first write.
|
||||
pub(crate) fn epoch() -> CultCacheEnvelope {
|
||||
HuginnMindEpoch::envelope(&schema_cache().unwrap()).unwrap()
|
||||
}
|
||||
|
||||
pub(crate) fn opened<S: MindStore>(store: S, instance: &str) -> Mind<S> {
|
||||
Mind::open_with(store, &slug(instance)).unwrap()
|
||||
}
|
||||
|
||||
/// Admits as Hands at the fixed clock.
|
||||
pub(crate) fn admit<S: MindStore>(mind: &mut Mind<S>, documents: Vec<PipelineDocument>) -> PipelineAdmissionOutcome {
|
||||
admit_at(mind, documents, now())
|
||||
}
|
||||
|
||||
/// Admits as Hands at the clock the caller names: the views order by the
|
||||
/// receipt's `committed_at`, which is this `now`.
|
||||
pub(crate) fn admit_at<S: MindStore>(
|
||||
mind: &mut Mind<S>,
|
||||
documents: Vec<PipelineDocument>,
|
||||
now: DateTime<Utc>,
|
||||
) -> PipelineAdmissionOutcome {
|
||||
let batch = PipelineAdmissionBatch { instance: mind.instance().clone(), provenance: provenance(Faculty::Hands), documents };
|
||||
mind.admit(batch, now)
|
||||
}
|
||||
|
||||
/// Admits as another faculty at the fixed clock: attribution the views filter
|
||||
/// on and no rule reads (ruling 18).
|
||||
pub(crate) fn admit_as<S: MindStore>(
|
||||
mind: &mut Mind<S>,
|
||||
faculty: Faculty,
|
||||
documents: Vec<PipelineDocument>,
|
||||
) -> PipelineAdmissionOutcome {
|
||||
let batch = PipelineAdmissionBatch { instance: mind.instance().clone(), provenance: provenance(faculty), documents };
|
||||
mind.admit(batch, now())
|
||||
}
|
||||
|
||||
/// The `n`th plain question, for the tests that need more of them than they
|
||||
/// need to tell apart.
|
||||
pub(crate) fn question_n(n: u32) -> PipelineDocument {
|
||||
question(&format!("Q{n}"), &["A", "B"], "A")
|
||||
}
|
||||
|
||||
/// The first write into a fresh yggdrasil mind: its identity, stewardship of
|
||||
/// the campaign repo, the campaign and a first target with one invariant.
|
||||
pub(crate) fn seed<S: MindStore>(mind: &mut Mind<S>) {
|
||||
let outcome = admit(mind, vec![instance(INSTANCE), stewardship(INSTANCE, REPO), campaign(&[REPO]), target(1, &[INVARIANT])]);
|
||||
assert!(matches!(outcome, PipelineAdmissionOutcome::Committed { .. }), "{outcome:?}");
|
||||
}
|
||||
|
||||
pub(crate) fn seeded() -> Mind<MemoryStore> {
|
||||
let mut mind = opened(MemoryStore::new(), INSTANCE);
|
||||
seed(&mut mind);
|
||||
mind
|
||||
}
|
||||
|
||||
pub(crate) fn refusal(outcome: PipelineAdmissionOutcome) -> MindRefusal {
|
||||
match outcome {
|
||||
PipelineAdmissionOutcome::Refused(refusal) => refusal,
|
||||
other => panic!("expected a refusal, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn committed(outcome: PipelineAdmissionOutcome) -> (String, Vec<PipelineRef>) {
|
||||
match outcome {
|
||||
PipelineAdmissionOutcome::Committed { receipt_id, writes, .. } => (receipt_id, writes),
|
||||
other => panic!("expected a commit, got {other:?}"),
|
||||
}
|
||||
}
|
||||
312
crates/huginn-mind/src/index.rs
Normal file
312
crates/huginn-mind/src/index.rs
Normal file
|
|
@ -0,0 +1,312 @@
|
|||
//! What of a document is searchable, and nothing else about the index.
|
||||
//!
|
||||
//! `index_text` is a rule about the leaf's documents: which fields are prose a
|
||||
//! reader would search for. It is a pure function of one document, so it reads
|
||||
//! no receipt, no provenance and no status, and nothing derived from it is
|
||||
//! stored: the daemon's projection recomputes it, and `INDEX_TEXT_VERSION` is
|
||||
//! how a change to it makes every projection rebuild. No network, no JSON, no
|
||||
//! store handle: the daemon owns the vector store and the embedder.
|
||||
|
||||
use epiphany_pipeline::{PipelineDocument, PipelineKind, PipelineRef};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::docs::Docs;
|
||||
use crate::mind::Mind;
|
||||
use crate::query::AdmissionIndex;
|
||||
use crate::refusal::MindRefusal;
|
||||
use crate::rows::root_and_local;
|
||||
use crate::store::MindStore;
|
||||
|
||||
/// Bumped whenever `index_text` changes what it returns for any document.
|
||||
pub const INDEX_TEXT_VERSION: u32 = 1;
|
||||
|
||||
/// The most bytes one document's text may carry, cut at a character boundary.
|
||||
pub const INDEX_TEXT_MAX_BYTES: usize = 16 * 1024;
|
||||
|
||||
/// One document as the projection needs it: the key, the searchable text and
|
||||
/// the digest of that text, and the ordinal of the receipt that wrote it.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct IndexEntry {
|
||||
pub id: PipelineRef,
|
||||
/// The key's root segment: a campaign slug, or an instance.
|
||||
pub root: String,
|
||||
pub text: String,
|
||||
pub text_sha256: String,
|
||||
pub ordinal: u64,
|
||||
}
|
||||
|
||||
/// The searchable text of a document, or `None` when its kind carries none
|
||||
/// (`instance`, `stewardship`) or every field it names is empty. Lines are
|
||||
/// joined by newlines. Withdrawn resolutions are indexed like any other: a
|
||||
/// closed thing stays findable.
|
||||
pub fn index_text(document: &PipelineDocument) -> Option<String> {
|
||||
use PipelineDocument as D;
|
||||
let mut lines: Vec<&str> = Vec::new();
|
||||
match document {
|
||||
D::Campaign(campaign) => lines.push(&campaign.title.0),
|
||||
D::Target(target) => {
|
||||
lines.extend(target.invariants.iter().map(|invariant| invariant.statement.0.as_str()));
|
||||
lines.extend(target.not_in_scope.iter().map(|line| line.0.as_str()));
|
||||
lines.extend(target.canonical_implementations.iter().map(|line| line.0.as_str()));
|
||||
}
|
||||
D::Question(question) => {
|
||||
lines.push(&question.title.0);
|
||||
lines.push(&question.question.0);
|
||||
lines.extend(question.options.iter().map(|option| option.text.0.as_str()));
|
||||
}
|
||||
D::Ruling(ruling) => {
|
||||
lines.push(&ruling.title.0);
|
||||
lines.push(&ruling.ruling.0);
|
||||
lines.extend(ruling.operator_quote.iter().map(|quote| quote.0.as_str()));
|
||||
}
|
||||
D::CutSpec(spec) => {
|
||||
lines.push(&spec.title.0);
|
||||
lines.extend(spec.first.iter().map(|line| line.0.as_str()));
|
||||
lines.extend(spec.adds.iter().map(|line| line.0.as_str()));
|
||||
lines.extend(spec.keeps_moves.iter().map(|line| line.0.as_str()));
|
||||
}
|
||||
D::CutReport(report) => {
|
||||
lines.extend(report.commits.iter().map(|commit| commit.subject.0.as_str()));
|
||||
for deviation in &report.deviations {
|
||||
lines.push(&deviation.what.0);
|
||||
lines.push(&deviation.why.0);
|
||||
}
|
||||
lines.extend(report.undone.iter().map(|line| line.0.as_str()));
|
||||
lines.extend(report.promises.iter().map(|promise| promise.text.0.as_str()));
|
||||
}
|
||||
D::Verdict(verdict) => lines.extend(verdict.claims.iter().map(|claim| claim.claim.0.as_str())),
|
||||
D::Finding(finding) => {
|
||||
lines.push(&finding.claim.0);
|
||||
lines.push(&finding.failure_scenario.0);
|
||||
lines.extend(finding.evidence.iter().map(|evidence| evidence.result.0.as_str()));
|
||||
}
|
||||
D::FollowUp(follow_up) => {
|
||||
lines.push(&follow_up.item.0);
|
||||
lines.push(&follow_up.why_it_can_wait.0);
|
||||
}
|
||||
D::Resolution(resolution) => lines.push(&resolution.rationale.0),
|
||||
D::HandOff(hand_off) => lines.push(&hand_off.reason.0),
|
||||
D::Instance(_) | D::Stewardship(_) => {}
|
||||
}
|
||||
lines.retain(|line| !line.is_empty());
|
||||
if lines.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let mut text = lines.join("\n");
|
||||
if text.len() > INDEX_TEXT_MAX_BYTES {
|
||||
let mut cut = INDEX_TEXT_MAX_BYTES;
|
||||
while !text.is_char_boundary(cut) {
|
||||
cut -= 1;
|
||||
}
|
||||
text.truncate(cut);
|
||||
}
|
||||
Some(text)
|
||||
}
|
||||
|
||||
fn sha256_hex(text: &str) -> String {
|
||||
Sha256::digest(text.as_bytes()).iter().map(|byte| format!("{byte:02x}")).collect()
|
||||
}
|
||||
|
||||
impl<S: MindStore> Mind<S> {
|
||||
/// The entries to project: every indexable document the mind holds, or
|
||||
/// only the named ones. The ordinal is the writing receipt's. A named
|
||||
/// document the mind does not hold is a refusal, not a silent omission: a
|
||||
/// caller names what a commit just landed.
|
||||
pub fn index_entries(&self, only: Option<&[PipelineRef]>) -> Result<Vec<IndexEntry>, MindRefusal> {
|
||||
let documents: Vec<(PipelineKind, String, PipelineDocument)> = match only {
|
||||
Some(refs) => {
|
||||
let mut held = Vec::with_capacity(refs.len());
|
||||
for reference in refs {
|
||||
reference.validate_ref()?;
|
||||
let document = self.get(reference.kind, &reference.id.0)?.ok_or_else(|| {
|
||||
MindRefusal::Unavailable { detail: format!("index: {} is not held", reference.id.0) }
|
||||
})?;
|
||||
held.push((reference.kind, reference.id.0.clone(), document));
|
||||
}
|
||||
held
|
||||
}
|
||||
None => Docs::from_image(self.envelopes())?
|
||||
.image
|
||||
.into_iter()
|
||||
.map(|held| (held.kind, held.key, held.document))
|
||||
.collect(),
|
||||
};
|
||||
let facts = AdmissionIndex::build(self)?;
|
||||
let mut entries = Vec::new();
|
||||
for (kind, key, document) in documents {
|
||||
let Some(text) = index_text(&document) else { continue };
|
||||
let ordinal = facts.ordinal_of(kind, &key)?;
|
||||
entries.push(IndexEntry {
|
||||
root: root_and_local(&key).0.to_string(),
|
||||
id: PipelineRef { kind, id: key.as_str().into() },
|
||||
text_sha256: sha256_hex(&text),
|
||||
text,
|
||||
ordinal,
|
||||
});
|
||||
}
|
||||
Ok(entries)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::fixtures::*;
|
||||
use crate::receipt::Faculty;
|
||||
use crate::store::test_stores::MemoryStore;
|
||||
use epiphany_pipeline::{
|
||||
ClaimOutcome, Deviation, FindingConfidence, PipelineDocument as D, PipelineKind as K, RulingAuthority,
|
||||
};
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
/// A mind holding one document of every kind that carries text, each with
|
||||
/// more than one text field where the kind has more than one, and the
|
||||
/// question's closure withdrawn beside them. Returns the hand-off's key.
|
||||
fn full_mind() -> (Mind<MemoryStore>, String) {
|
||||
let mut mind = opened(MemoryStore::new(), INSTANCE);
|
||||
let D::Target(mut target_doc) = target(1, &[INVARIANT, "second"]) else { panic!() };
|
||||
target_doc.not_in_scope = vec!["No client writes.".into()];
|
||||
target_doc.canonical_implementations = vec!["huginn-mind admission".into()];
|
||||
committed(admit(
|
||||
&mut mind,
|
||||
vec![instance(INSTANCE), stewardship(INSTANCE, REPO), campaign(&[REPO]), D::Target(target_doc)],
|
||||
));
|
||||
committed(admit(&mut mind, vec![question("Q1", &["A", "B"], "A")]));
|
||||
let mut ruling_doc = ruling("R1");
|
||||
ruling_doc.answers = Some(s(&id("question", "Q1")));
|
||||
ruling_doc.choice = Some(l("A"));
|
||||
ruling_doc.operator_quote = Some("Own it.".into());
|
||||
ruling_doc.authority = RulingAuthority::Operator;
|
||||
committed(admit(&mut mind, vec![D::Ruling(ruling_doc)]));
|
||||
// The derived closure of Q1 is withdrawn: a withdrawn resolution is a
|
||||
// document like any other and stays findable.
|
||||
let closed = id("resolution", "question.Q1.n1");
|
||||
committed(admit(&mut mind, vec![resolution(r(K::Resolution, &closed), withdrawn())]));
|
||||
let mut spec_doc = cut_spec("1", 1);
|
||||
spec_doc.first = vec!["Read the leaf.".into()];
|
||||
spec_doc.adds = vec!["An index module.".into(), "A worker.".into()];
|
||||
spec_doc.keeps_moves = vec!["Admission stays.".into()];
|
||||
committed(admit(&mut mind, vec![D::CutSpec(spec_doc)]));
|
||||
let mut report_doc = cut_report("1", 1);
|
||||
report_doc.deviations = vec![Deviation { what: "Changed a name.".into(), why: "It collided.".into() }];
|
||||
report_doc.undone = vec!["The live smoke.".into()];
|
||||
committed(admit(
|
||||
&mut mind,
|
||||
vec![D::CutReport(report_doc), follow_up("FU-1", r(K::Question, &id("question", "Q1")))],
|
||||
));
|
||||
committed(admit(
|
||||
&mut mind,
|
||||
vec![
|
||||
verdict("1", 1, vec![claim(ClaimOutcome::Falsified, &[&id("finding", "cut-1.s1.F1")], Some("P1"), &["M1"])]),
|
||||
D::Finding(finding("1", 1, "F1", FindingConfidence::Confirmed)),
|
||||
],
|
||||
));
|
||||
let (_, writes) = committed(admit(&mut mind, vec![hand_off(INSTANCE, OTHER_INSTANCE, REPO, &[])]));
|
||||
let hand_off_key = writes.iter().find(|write| write.kind == K::HandOff).unwrap().id.0.clone();
|
||||
(mind, hand_off_key)
|
||||
}
|
||||
|
||||
fn texts(mind: &Mind<MemoryStore>) -> BTreeMap<String, String> {
|
||||
mind.index_entries(None).unwrap().into_iter().map(|entry| (entry.id.id.0, entry.text)).collect()
|
||||
}
|
||||
|
||||
/// Each kind's search text is its named fields, joined by newlines, and
|
||||
/// the two kinds with no prose have no entry.
|
||||
#[test]
|
||||
fn index_text_is_each_kinds_named_fields_and_the_identity_kinds_have_none() {
|
||||
let (mind, hand_off_key) = full_mind();
|
||||
let expected: Vec<(String, &str)> = vec![
|
||||
(id("campaign", "self"), "Eureka pipeline state"),
|
||||
(id("target", "r1"), "Only admission writes.\nOnly admission writes.\nNo client writes.\nhuginn-mind admission"),
|
||||
(id("question", "Q1"), "Who owns the state?\nWho owns the state?\nan option\nan option"),
|
||||
(id("ruling", "R1"), "An instance owns its mind\nAn instance owns its mind.\nOwn it."),
|
||||
(id("cut_spec", "cut-1.r1"), "A cut\nRead the leaf.\nAn index module.\nA worker.\nAdmission stays."),
|
||||
(
|
||||
id("cut_report", "cut-1.h1"),
|
||||
"Land the cut\nChanged a name.\nIt collided.\nThe live smoke.\nOne derived key per document.",
|
||||
),
|
||||
(id("verdict", "cut-1.s1"), "Falsified claim"),
|
||||
(id("finding", "cut-1.s1.F1"), "A dotted label composes two keys.\nTwo documents claim one key.\nok"),
|
||||
(id("follow_up", "FU-1"), "Per-kind admission rules.\nThe organ owns admission."),
|
||||
// The withdrawn closure of Q1, admitted by hand.
|
||||
(id("resolution", "resolution.question.Q1.n1.n1"), "Resolved."),
|
||||
(hand_off_key, "The workstation mind takes the campaign."),
|
||||
];
|
||||
let held = texts(&mind);
|
||||
for (key, text) in &expected {
|
||||
match held.get(key) {
|
||||
Some(found) => assert_eq!(found, text, "{key}"),
|
||||
None => panic!("{key} has no entry; entries are {:?}", held.keys().collect::<Vec<_>>()),
|
||||
}
|
||||
}
|
||||
let kinds: Vec<K> = mind.index_entries(None).unwrap().iter().map(|entry| entry.id.kind).collect();
|
||||
assert!(!kinds.contains(&K::Instance) && !kinds.contains(&K::Stewardship), "{kinds:?}");
|
||||
assert!(kinds.contains(&K::Resolution), "the derived closures are indexed too");
|
||||
}
|
||||
|
||||
/// The text is the document's alone: the same document admitted by another
|
||||
/// faculty at another clock, or closed by a later ruling, has the same
|
||||
/// text and digest. Only the ordinal is the receipt's.
|
||||
#[test]
|
||||
fn index_text_ignores_admission_facts_and_indexes_withdrawn_resolutions() {
|
||||
let mut mind = seeded();
|
||||
committed(admit(&mut mind, vec![question("Q1", &["A", "B"], "A")]));
|
||||
let q1 = id("question", "Q1");
|
||||
let before = mind.index_entries(Some(&[r(K::Question, &q1)])).unwrap();
|
||||
|
||||
let mut other = seeded();
|
||||
committed(admit_as(&mut other, Faculty::Soul, vec![question("Q1", &["A", "B"], "A")]));
|
||||
let mut later = ruling("R1");
|
||||
later.answers = Some(s(&q1));
|
||||
later.choice = Some(l("A"));
|
||||
committed(admit(&mut other, vec![D::Ruling(later)]));
|
||||
let after = other.index_entries(Some(&[r(K::Question, &q1)])).unwrap();
|
||||
assert_eq!(before, after, "a closed question is the same entry as an open one");
|
||||
assert_eq!(before[0].ordinal, 2);
|
||||
|
||||
let closure = id("resolution", "question.Q1.n1");
|
||||
committed(admit(&mut other, vec![resolution(r(K::Resolution, &closure), withdrawn())]));
|
||||
let withdrawn_id = id("resolution", "resolution.question.Q1.n1.n1");
|
||||
let entry = other.index_entries(Some(&[r(K::Resolution, &withdrawn_id)])).unwrap();
|
||||
assert_eq!(entry.len(), 1, "a withdrawn resolution is indexed");
|
||||
assert_eq!(entry[0].text, "Resolved.");
|
||||
assert_eq!(entry[0].root, CAMPAIGN);
|
||||
assert_eq!(entry[0].ordinal, 4);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_digest_is_sha256_of_the_text_and_a_missing_document_is_refused() {
|
||||
let mind = seeded();
|
||||
let campaign_ref = r(K::Campaign, &id("campaign", "self"));
|
||||
let entry = &mind.index_entries(Some(std::slice::from_ref(&campaign_ref))).unwrap()[0];
|
||||
assert_eq!(entry.text_sha256, "49c7e65adaad4bd34261300e9c5575847357b8f269508c9b2a46ec2aa75cc019");
|
||||
assert_eq!((entry.root.as_str(), entry.ordinal), (CAMPAIGN, 1));
|
||||
let absent = r(K::Question, &id("question", "Q9"));
|
||||
assert!(matches!(mind.index_entries(Some(&[absent])), Err(MindRefusal::Unavailable { .. })));
|
||||
// The identity and the stewardship carry no text: the seeded mind's
|
||||
// indexable documents are its campaign and its target.
|
||||
let all = mind.index_entries(None).unwrap();
|
||||
assert_eq!(all.len(), 2, "{all:?}");
|
||||
}
|
||||
|
||||
/// The bound is bytes, cut at a character boundary: three-byte characters
|
||||
/// put the byte limit inside one.
|
||||
#[test]
|
||||
fn text_is_bounded_at_a_character_boundary() {
|
||||
let wide: String = format!("{}x", "\u{20ac}".repeat(333));
|
||||
assert_eq!(wide.len(), 1000);
|
||||
let mut spec = cut_spec("1", 1);
|
||||
spec.adds = (0..40).map(|_| wide.as_str().into()).collect();
|
||||
let text = index_text(&D::CutSpec(spec)).unwrap();
|
||||
// The title and its newline, then full lines of 1,001 bytes, then part
|
||||
// of the next one, cut back to the last whole character.
|
||||
let head = "A cut\n".len();
|
||||
let stride = 1001;
|
||||
let full_lines = (INDEX_TEXT_MAX_BYTES - head) / stride;
|
||||
let consumed = head + full_lines * stride;
|
||||
let inside = INDEX_TEXT_MAX_BYTES - consumed;
|
||||
assert_ne!(inside % 3, 0, "the limit falls inside a character");
|
||||
assert_eq!(text.len(), consumed + inside - inside % 3);
|
||||
assert!(text.ends_with('\u{20ac}'));
|
||||
}
|
||||
}
|
||||
62
crates/huginn-mind/src/lib.rs
Normal file
62
crates/huginn-mind/src/lib.rs
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
//! `huginn-mind`: one instance's mind as typed CultCache state.
|
||||
//!
|
||||
//! The crate owns three decisions and nothing else. `Mind::open` owns "may
|
||||
//! this store be this instance's mind": the opener refuses a foreign type, a
|
||||
//! foreign epoch, a missing identity or a foreign identity before it attaches
|
||||
//! anything, and takes the store's per-path lock first. `receipt::commit` owns "did a batch enter, whole, with a
|
||||
//! receipt": one compare-and-swap over the store, a receipt naming the exact
|
||||
//! bytes read and written, and a typed conflict when the swap loses.
|
||||
//! `Mind::admit_prepared` owns "may this batch enter": every cross-field and
|
||||
//! cross-document rule, every derived write, and the replay check, in one
|
||||
//! ordered path that `admit`, the daemon's sink and the hand-off all share.
|
||||
//!
|
||||
//! The read side derives status, joins the receipts and answers one typed
|
||||
//! selection (CultNet's, over rows this crate supplies) through the same
|
||||
//! `docs` the rules use; nothing is stored for it.
|
||||
//!
|
||||
//! The wire vocabulary and its operation envelope (`envelope`) are here too,
|
||||
//! so the daemon and the client share one set
|
||||
//! of types.
|
||||
//!
|
||||
//! Document shape, bounds, formats and keys are `epiphany-pipeline`'s; the
|
||||
//! organ registers, prepares, decodes and validates through the leaf's four
|
||||
//! doors and never re-derives a key. The store is CultLib's owned redb
|
||||
//! CultCache, whose lifetime-long exclusive lock is the single-writer
|
||||
//! invariant's mechanism. No function here reads a clock or the environment:
|
||||
//! `now` is passed in.
|
||||
|
||||
pub mod admission;
|
||||
mod docs;
|
||||
pub mod envelope;
|
||||
pub mod index;
|
||||
pub mod mind;
|
||||
mod rows;
|
||||
pub mod query;
|
||||
pub mod receipt;
|
||||
pub mod refusal;
|
||||
pub mod store;
|
||||
pub mod wire;
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod fixtures;
|
||||
|
||||
pub use docs::CitationRole;
|
||||
pub use index::{INDEX_TEXT_MAX_BYTES, INDEX_TEXT_VERSION, IndexEntry, index_text};
|
||||
pub use admission::{BATCH_MAX, PipelineAdmissionBatch, PipelineAdmissionOutcome};
|
||||
pub use mind::{HuginnMindEpoch, Mind};
|
||||
pub use query::{
|
||||
AdmissionFacts, PipelineDocumentSummary, PipelineDocumentView, PipelineEdge, PipelineFacts, PipelinePageItems,
|
||||
PipelineSelectionPage, PipelineStatus, PipelineStatusSummary, Ranked, SUMMARY_MAX_BYTES, SemanticQuery,
|
||||
};
|
||||
pub use receipt::{DocumentVersion, Faculty, HuginnCommitReceipt, PipelineProvenance, RECEIPT_SCHEMA_VERSION};
|
||||
pub use refusal::MindRefusal;
|
||||
pub use store::{MindStore, OwnedRedbMessagePackBackingStore};
|
||||
pub use wire::{
|
||||
DeferredAnswer, HuginnMindRequest, HuginnMindResponse, IndexStatus, MAX_DEFERRED_BODY_BYTES, MIND_REQUEST_SCHEMA, MIND_REQUEST_SCHEMA_JSON,
|
||||
MIND_RESPONSE_SCHEMA, MIND_RESPONSE_SCHEMA_JSON, MIND_SERVICE_ID, MindStatus,
|
||||
};
|
||||
|
||||
/// The leaf, whole, through the one crate that pins its rev: the daemon and
|
||||
/// the client name `Slug`, `PipelineRef` and the document types from here, so
|
||||
/// one `[dependencies]` entry decides which revision the workspace speaks.
|
||||
pub use epiphany_pipeline;
|
||||
781
crates/huginn-mind/src/mind.rs
Normal file
781
crates/huginn-mind/src/mind.rs
Normal file
|
|
@ -0,0 +1,781 @@
|
|||
//! The mind: one instance's store, opened fail-closed, and its image.
|
||||
//!
|
||||
//! Ruling 14: a store is canonical to exactly one instance, and the identity
|
||||
//! lives in the state, in the `instance` document, not in a path. The path
|
||||
//! under the state root is derived from the slug for convenience; the
|
||||
//! document is the authority, and a store moved to another instance's
|
||||
//! directory is refused. Ruling 15: one writer, the owned redb store's
|
||||
//! lifetime-long exclusive lock. Ruling 20's analogue: the opener validates
|
||||
//! every gate before it attaches, so a refused store is never read into an
|
||||
//! image.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use cultcache_rs::{CultCache, CultCacheEnvelope, DatabaseEntry, OwnedRedbMessagePackBackingStore};
|
||||
use cultnet_rs::CursorKey;
|
||||
use epiphany_pipeline::{PIPELINE_SCHEMA_EPOCH, PipelineDocument, PipelineKind, Slug, register_pipeline_document_types};
|
||||
|
||||
use crate::receipt::HuginnCommitReceipt;
|
||||
use crate::refusal::MindRefusal;
|
||||
use crate::store::MindStore;
|
||||
|
||||
/// The store's record of the schema epoch it was written at, keyed by the
|
||||
/// epoch string. Derived by admission on the first write; read by the opener,
|
||||
/// which refuses a store written at any other epoch (`ForeignEpoch`) so a
|
||||
/// breaking schema bump refuses the old store instead of misreading it.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
|
||||
#[cultcache(type = "huginn.mind_epoch.v1", schema = "HuginnMindEpoch")]
|
||||
pub struct HuginnMindEpoch {
|
||||
#[cultcache(key = 0)]
|
||||
pub schema_epoch: String,
|
||||
}
|
||||
|
||||
impl HuginnMindEpoch {
|
||||
/// The one envelope the record ever has: the current epoch, keyed by
|
||||
/// itself. Admission appends it to the first write.
|
||||
pub(crate) fn envelope(cache: &CultCache) -> Result<CultCacheEnvelope, MindRefusal> {
|
||||
let record = Self { schema_epoch: PIPELINE_SCHEMA_EPOCH.to_string() };
|
||||
cache
|
||||
.prepare_entry_named(PIPELINE_SCHEMA_EPOCH, &record)
|
||||
.map(|(envelope, _)| envelope)
|
||||
.map_err(unavailable)
|
||||
}
|
||||
}
|
||||
|
||||
/// A store or cache error, carried as data.
|
||||
pub(crate) fn unavailable(error: anyhow::Error) -> MindRefusal {
|
||||
MindRefusal::Unavailable { detail: format!("{error:#}") }
|
||||
}
|
||||
|
||||
/// A declared name's grammar, checked before its bytes are compared to
|
||||
/// anything or reach the filesystem. `Slug` is `epiphany_pipeline`'s,
|
||||
/// dot-joined ASCII labels, and the leaf now opens a public door onto its own
|
||||
/// check, `Slug::validate_slug`, on the pattern of `PipelineRef::validate_ref`
|
||||
/// (Self's ruling on the F6 fork): the grammar is the leaf's, never
|
||||
/// re-derived here, so a fold that widens what counts as ASCII (a fullwidth
|
||||
/// character folded to its plain form, say) has nothing local to weaken.
|
||||
///
|
||||
/// The leaf's own refusal names its own field, `slug`, and for a dotted name
|
||||
/// the one label that failed rather than the whole declared name; a caller
|
||||
/// here reports neither, since both would misname what the caller actually
|
||||
/// sent. This instead names the field the declared name was held under and
|
||||
/// the whole declared name, so `require_instance` and `Mind::open` each
|
||||
/// refuse by the name a client can recognise, not `instance.instance` (the
|
||||
/// old wrapper's field, a client never sent) and not a label fragment.
|
||||
fn require_grammatical_slug(field: &str, declared: &Slug) -> Result<(), MindRefusal> {
|
||||
declared.validate_slug().map_err(|_| {
|
||||
MindRefusal::Document(epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: field.into(),
|
||||
value: declared.0.clone(),
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/// A cache that knows the fifteen types a mind's store may hold: the leaf's
|
||||
/// thirteen through its registrar, the epoch record and the commit receipt.
|
||||
/// It is the cache every envelope is prepared against.
|
||||
pub(crate) fn schema_cache() -> Result<CultCache, MindRefusal> {
|
||||
let mut cache = CultCache::new();
|
||||
register_pipeline_document_types(&mut cache).map_err(unavailable)?;
|
||||
cache.register_entry_type::<HuginnMindEpoch>().map_err(unavailable)?;
|
||||
cache.register_entry_type::<HuginnCommitReceipt>().map_err(unavailable)?;
|
||||
Ok(cache)
|
||||
}
|
||||
|
||||
fn is_known_type(type_id: &str) -> bool {
|
||||
type_id == HuginnMindEpoch::TYPE
|
||||
|| type_id == HuginnCommitReceipt::TYPE
|
||||
|| PipelineKind::ALL.iter().any(|kind| kind.type_id() == type_id)
|
||||
}
|
||||
|
||||
/// One instance's mind: the declared instance, the store, the registered
|
||||
/// cache attached to it, and the image (the store's envelopes as last pulled,
|
||||
/// in identity order). The image is a cache of the store and is re-pulled
|
||||
/// after every commit; nothing writes it directly.
|
||||
pub struct Mind<S: MindStore> {
|
||||
instance: Slug,
|
||||
store: S,
|
||||
cache: CultCache,
|
||||
image: Vec<CultCacheEnvelope>,
|
||||
/// Mints and verifies this process's selection cursors. Never stored: a
|
||||
/// cursor does not survive a reopen, which the substrate accepts.
|
||||
cursor_key: CursorKey,
|
||||
}
|
||||
|
||||
impl Mind<OwnedRedbMessagePackBackingStore> {
|
||||
/// `<state_root>/minds/<instance>/mind.redb`. Derived for convenience;
|
||||
/// the `instance` document inside is the identity. Private: every caller
|
||||
/// that can reach a raw path must go through the grammar door first, so
|
||||
/// this is never exposed on its own. `open` calls it only after
|
||||
/// `require_grammatical_slug` has already passed.
|
||||
fn path_for(state_root: &Path, instance: &Slug) -> PathBuf {
|
||||
state_root.join("minds").join(&instance.0).join("mind.redb")
|
||||
}
|
||||
|
||||
/// Test-only door onto the same path: validates through
|
||||
/// `require_grammatical_slug` first, exactly as `open` does, so a test
|
||||
/// that must reach the raw store directly (to break a row on purpose)
|
||||
/// cannot bypass the grammar the way the old public `path_for` did. Built
|
||||
/// only for `cfg(test)` in this crate and for `huginn-daemon`'s own tests
|
||||
/// via the `test-support` feature; never compiled into a release binary.
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub fn store_path_for(state_root: &Path, instance: &Slug) -> Result<PathBuf, MindRefusal> {
|
||||
require_grammatical_slug("instance", instance)?;
|
||||
Ok(Self::path_for(state_root, instance))
|
||||
}
|
||||
|
||||
/// Opens the instance's mind under the state root, taking the store's
|
||||
/// exclusive lock for the mind's lifetime. A second owner of the same
|
||||
/// path, in this process or another, is `MindAlreadyOwned`. An empty
|
||||
/// store is a valid open; only the first admission may write into it.
|
||||
pub fn open(state_root: &Path, instance: &Slug) -> Result<Self, MindRefusal> {
|
||||
require_grammatical_slug("instance", instance)?;
|
||||
let path = Self::path_for(state_root, instance);
|
||||
let store = OwnedRedbMessagePackBackingStore::new(&path).map_err(|error| {
|
||||
// The store reports a held lock as an error like any other; its
|
||||
// message is the only signal, so it is matched here, once.
|
||||
if format!("{error:#}").contains("already has an active owner") {
|
||||
MindRefusal::MindAlreadyOwned { path: path.display().to_string() }
|
||||
} else {
|
||||
unavailable(error)
|
||||
}
|
||||
})?;
|
||||
Self::open_checked(store, instance)
|
||||
}
|
||||
}
|
||||
|
||||
impl<S: MindStore> Mind<S> {
|
||||
/// Test-only door onto the same fail-closed sequence `open` runs, for a
|
||||
/// store already in hand (planted directly, or opened by a test through
|
||||
/// `store_path_for`). Gated exactly like `store_path_for`: it compiles
|
||||
/// only under `cfg(test)` in this crate or for a dependent's own tests
|
||||
/// via the `test-support` feature. `huginn-daemon` enables that feature
|
||||
/// only in `[dev-dependencies]`, so a plain `cargo build --release -p
|
||||
/// huginn-daemon` never pulls it into the shipped binary; but
|
||||
/// `--all-targets` (and `cargo test`) unifies dev-dependency features
|
||||
/// across the whole invocation, and that build's `huginn-daemon` binary
|
||||
/// does carry `open_with`. As of this writing no Idunn deploy recipe for
|
||||
/// Huginn exists to pin which of those commands is the actual shipped
|
||||
/// build, so treat this as the open question rather than a settled
|
||||
/// release-only guarantee. A second in-process owner of one store is not
|
||||
/// yet sealed for every `MindStore` impl (F2's recorded follow-up);
|
||||
/// production reaches this sequence only through `Mind::open`, which
|
||||
/// takes the per-path redb lock first.
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub fn open_with(store: S, instance: &Slug) -> Result<Self, MindRefusal> {
|
||||
Self::open_checked(store, instance)
|
||||
}
|
||||
|
||||
/// Fail-closed, in this order, nothing attached until every step passes:
|
||||
/// pull the raw envelopes; if anything is stored, exactly one epoch
|
||||
/// record at the current epoch; every type is one of the fifteen; and
|
||||
/// exactly one `instance` document naming the declared instance; then
|
||||
/// register, attach and pull. The epoch gate runs first so a store
|
||||
/// written at a foreign epoch is refused as `ForeignEpoch`, never as
|
||||
/// `ForeignStore`: a real epoch bump always moves every type id, so the
|
||||
/// type gate would otherwise fire first and the epoch gate would never
|
||||
/// be reached for the one case it exists for (F5). This is the one
|
||||
/// construction site of a `Mind`; `open` and the test-only `open_with`
|
||||
/// both call it and neither reimplements it.
|
||||
fn open_checked(store: S, instance: &Slug) -> Result<Self, MindRefusal> {
|
||||
require_grammatical_slug("instance", instance)?;
|
||||
let raw = store.pull_all().map_err(unavailable)?;
|
||||
refuse_foreign_epoch(&raw)?;
|
||||
refuse_foreign_types(&raw)?;
|
||||
refuse_foreign_identity(&raw, instance)?;
|
||||
let (cache, image) = attach(store.clone())?;
|
||||
Ok(Self { instance: instance.clone(), store, cache, image, cursor_key: CursorKey::random() })
|
||||
}
|
||||
|
||||
pub fn instance(&self) -> &Slug {
|
||||
&self.instance
|
||||
}
|
||||
|
||||
/// Ruling 14 across every transport: the declared instance is this mind's,
|
||||
/// else `ForeignInstance { declared, mind }`. A1 for admission; the daemon
|
||||
/// asks it for every read that names an instance, and compares nothing
|
||||
/// itself.
|
||||
pub fn require_instance(&self, declared: &Slug) -> Result<(), MindRefusal> {
|
||||
require_grammatical_slug("declared", declared)?;
|
||||
if declared != self.instance() {
|
||||
return Err(MindRefusal::ForeignInstance {
|
||||
declared: declared.0.clone(),
|
||||
mind: self.instance().0.clone(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The image: every envelope the store held at the last pull, in
|
||||
/// identity order. Cut 10's snapshot source reads this.
|
||||
pub fn envelopes(&self) -> &[CultCacheEnvelope] {
|
||||
&self.image
|
||||
}
|
||||
|
||||
pub(crate) fn cursor_key(&self) -> &CursorKey {
|
||||
&self.cursor_key
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.image.is_empty()
|
||||
}
|
||||
|
||||
/// The stored bytes of one document, by kind and id.
|
||||
pub fn envelope(&self, kind: PipelineKind, id: &str) -> Option<&CultCacheEnvelope> {
|
||||
self.raw_envelope(kind.type_id(), id)
|
||||
}
|
||||
|
||||
/// One document, decoded through the leaf.
|
||||
pub fn get(&self, kind: PipelineKind, id: &str) -> Result<Option<PipelineDocument>, MindRefusal> {
|
||||
self.envelope(kind, id)
|
||||
.map(|envelope| PipelineDocument::decode(envelope).map_err(MindRefusal::Document))
|
||||
.transpose()
|
||||
}
|
||||
|
||||
/// Every commit receipt in the image, in receipt-id order.
|
||||
pub(crate) fn receipts(&self) -> Result<Vec<HuginnCommitReceipt>, MindRefusal> {
|
||||
self.image
|
||||
.iter()
|
||||
.filter(|envelope| envelope.r#type == HuginnCommitReceipt::TYPE)
|
||||
.map(|envelope| {
|
||||
rmp_serde::from_slice::<HuginnCommitReceipt>(&envelope.payload).map_err(|error| {
|
||||
MindRefusal::Unavailable { detail: format!("receipt {} does not decode: {error}", envelope.key) }
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The mind's own identity: the receipt id of its first admission
|
||||
/// (ordinal 1), which admission never rewrites and which digests the
|
||||
/// instance and the first batch's bytes. `None` for a mind that has not
|
||||
/// been written yet. The instance name is only a label, since two stores
|
||||
/// on two hosts may share it; this tells them apart unless their first
|
||||
/// batches were byte-identical.
|
||||
pub fn genesis_receipt_id(&self) -> Result<Option<String>, MindRefusal> {
|
||||
Ok(self.receipts()?.into_iter().find(|receipt| receipt.ordinal == 1).map(|receipt| receipt.receipt_id))
|
||||
}
|
||||
|
||||
pub(crate) fn raw_envelope(&self, type_id: &str, key: &str) -> Option<&CultCacheEnvelope> {
|
||||
self.image.iter().find(|envelope| envelope.r#type == type_id && envelope.key == key)
|
||||
}
|
||||
|
||||
pub(crate) fn cache(&self) -> &CultCache {
|
||||
&self.cache
|
||||
}
|
||||
|
||||
pub(crate) fn store(&self) -> &S {
|
||||
&self.store
|
||||
}
|
||||
|
||||
/// Re-pulls the image from the store through the attached cache.
|
||||
pub(crate) fn refresh(&mut self) -> Result<(), MindRefusal> {
|
||||
self.cache.pull_all_backing_stores().map_err(unavailable)?;
|
||||
self.image = snapshot(&self.cache);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Step 3: a runtime store, a Mind store or any other file passed by mistake
|
||||
/// dies on its first foreign type.
|
||||
fn refuse_foreign_types(raw: &[CultCacheEnvelope]) -> Result<(), MindRefusal> {
|
||||
if let Some(foreign) = raw.iter().find(|envelope| !is_known_type(&envelope.r#type)) {
|
||||
return Err(MindRefusal::ForeignStore { r#type: foreign.r#type.clone() });
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Step 2: a non-empty store carries exactly one epoch record, keyed by the
|
||||
/// epoch it names, at the epoch this binary writes. A store with no epoch
|
||||
/// record at all is `MissingIdentity`; every other defect is `ForeignEpoch`,
|
||||
/// whose `found` names the defect: the stored epoch when the value is
|
||||
/// foreign, the record's key when a record is keyed by anything else, and the
|
||||
/// record count rendered as `"<n> records"` when more than one is stored. The
|
||||
/// count is decided before any value is read, so a current record beside a
|
||||
/// foreign one refuses the same way whichever the store returns first.
|
||||
fn refuse_foreign_epoch(raw: &[CultCacheEnvelope]) -> Result<(), MindRefusal> {
|
||||
if raw.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let foreign = |found: String| MindRefusal::ForeignEpoch { found, expected: PIPELINE_SCHEMA_EPOCH.into() };
|
||||
let mut records = raw.iter().filter(|envelope| envelope.r#type == HuginnMindEpoch::TYPE);
|
||||
let Some(record) = records.next() else {
|
||||
return Err(MindRefusal::MissingIdentity);
|
||||
};
|
||||
let extra = records.count();
|
||||
if extra > 0 {
|
||||
return Err(foreign(format!("{} records", extra + 1)));
|
||||
}
|
||||
if record.key != PIPELINE_SCHEMA_EPOCH {
|
||||
return Err(foreign(record.key.clone()));
|
||||
}
|
||||
let record: HuginnMindEpoch = rmp_serde::from_slice(&record.payload)
|
||||
.map_err(|error| MindRefusal::Unavailable { detail: format!("epoch record does not decode: {error}") })?;
|
||||
if record.schema_epoch != PIPELINE_SCHEMA_EPOCH {
|
||||
return Err(foreign(record.schema_epoch));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Step 4: a non-empty store carries exactly one `instance` document, and it
|
||||
/// names the declared instance. This is where a store moved to another
|
||||
/// instance's directory is refused.
|
||||
fn refuse_foreign_identity(raw: &[CultCacheEnvelope], instance: &Slug) -> Result<(), MindRefusal> {
|
||||
if raw.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let mut documents = raw.iter().filter(|envelope| envelope.r#type == PipelineKind::Instance.type_id());
|
||||
let (Some(envelope), None) = (documents.next(), documents.next()) else {
|
||||
return Err(MindRefusal::MissingIdentity);
|
||||
};
|
||||
let PipelineDocument::Instance(stored) = PipelineDocument::decode(envelope)? else {
|
||||
return Err(MindRefusal::MissingIdentity);
|
||||
};
|
||||
if stored.instance != *instance {
|
||||
return Err(MindRefusal::ForeignInstance { declared: instance.0.clone(), mind: stored.instance.0 });
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Step 5: the registered cache, the store as its one generic home, and the
|
||||
/// image pulled through it.
|
||||
fn attach<S: MindStore>(store: S) -> Result<(CultCache, Vec<CultCacheEnvelope>), MindRefusal> {
|
||||
let mut cache = schema_cache()?;
|
||||
cache.add_generic_backing_store(store).map_err(unavailable)?;
|
||||
cache.pull_all_backing_stores().map_err(unavailable)?;
|
||||
let image = snapshot(&cache);
|
||||
Ok((cache, image))
|
||||
}
|
||||
|
||||
fn snapshot(cache: &CultCache) -> Vec<CultCacheEnvelope> {
|
||||
let mut image = cache.snapshot_envelopes();
|
||||
image.sort_by(|left, right| (&left.r#type, &left.key).cmp(&(&right.r#type, &right.key)));
|
||||
image
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::fixtures::{INSTANCE, epoch, instance, prepare, slug};
|
||||
use crate::store::test_stores::MemoryStore;
|
||||
use cultcache_rs::CacheBackingStore;
|
||||
|
||||
fn planted(rows: Vec<CultCacheEnvelope>) -> MemoryStore {
|
||||
let store = MemoryStore::new();
|
||||
for row in rows {
|
||||
store.plant(row);
|
||||
}
|
||||
store
|
||||
}
|
||||
|
||||
fn foreign(type_id: &str) -> CultCacheEnvelope {
|
||||
let mut envelope = epoch();
|
||||
envelope.r#type = type_id.into();
|
||||
envelope
|
||||
}
|
||||
|
||||
const FOREIGN_EPOCH: &str = "epiphany.pipeline.epoch.v0";
|
||||
|
||||
/// A type id that is foreign under every epoch, unlike
|
||||
/// `"epiphany.pipeline.campaign.v1"`: that string names the pinned
|
||||
/// leaf's real, currently-registered campaign kind, so `is_known_type`
|
||||
/// accepts it and a gate-order mutant that runs the type gate first
|
||||
/// still passes these fixtures by accident. `_never_a_kind_` cannot
|
||||
/// collide with any real `PipelineKind` id at any epoch this leaf pin
|
||||
/// will ever carry, so these fixtures actually exercise the epoch-first
|
||||
/// order rather than getting lucky on a real id.
|
||||
const NEVER_A_KNOWN_TYPE: &str = "epiphany.pipeline._never_a_kind_.v1";
|
||||
|
||||
/// An epoch record under an arbitrary key naming an arbitrary epoch, so a
|
||||
/// key defect and a value defect can be planted apart.
|
||||
fn epoch_record(key: &str, schema_epoch: &str) -> CultCacheEnvelope {
|
||||
let cache = schema_cache().unwrap();
|
||||
let record = HuginnMindEpoch { schema_epoch: schema_epoch.into() };
|
||||
cache.prepare_entry_named(key, &record).unwrap().0
|
||||
}
|
||||
|
||||
fn foreign_epoch() -> CultCacheEnvelope {
|
||||
epoch_record(FOREIGN_EPOCH, FOREIGN_EPOCH)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_opener_refuses_foreign_epoch_missing_identity_and_foreign_type_before_attaching() {
|
||||
let yggdrasil = slug(INSTANCE);
|
||||
let cases: Vec<(Vec<CultCacheEnvelope>, MindRefusal)> = vec![
|
||||
// Step 3, reached because the epoch record here is current: a
|
||||
// runtime store passed by mistake.
|
||||
(
|
||||
vec![foreign("epiphany.runtime_spine.v47"), epoch(), prepare(&instance(INSTANCE))],
|
||||
MindRefusal::ForeignStore { r#type: "epiphany.runtime_spine.v47".into() },
|
||||
),
|
||||
// Step 2 before step 4: a foreign epoch with no identity at all.
|
||||
// Its key is foreign too, which is what refuses it here.
|
||||
(
|
||||
vec![foreign_epoch()],
|
||||
MindRefusal::ForeignEpoch { found: FOREIGN_EPOCH.into(), expected: PIPELINE_SCHEMA_EPOCH.into() },
|
||||
),
|
||||
// Step 2: the value, under the one key the record may have.
|
||||
(
|
||||
vec![epoch_record(PIPELINE_SCHEMA_EPOCH, FOREIGN_EPOCH), prepare(&instance(INSTANCE))],
|
||||
MindRefusal::ForeignEpoch { found: FOREIGN_EPOCH.into(), expected: PIPELINE_SCHEMA_EPOCH.into() },
|
||||
),
|
||||
// Step 2: the current epoch under any other key is not the epoch
|
||||
// record, whatever it says about itself.
|
||||
(
|
||||
vec![epoch_record("not-the-epoch", PIPELINE_SCHEMA_EPOCH), prepare(&instance(INSTANCE))],
|
||||
MindRefusal::ForeignEpoch { found: "not-the-epoch".into(), expected: PIPELINE_SCHEMA_EPOCH.into() },
|
||||
),
|
||||
// Step 2: two records are no record, counted before either value
|
||||
// is read.
|
||||
(
|
||||
vec![epoch(), epoch_record("second", PIPELINE_SCHEMA_EPOCH), prepare(&instance(INSTANCE))],
|
||||
MindRefusal::ForeignEpoch { found: "2 records".into(), expected: PIPELINE_SCHEMA_EPOCH.into() },
|
||||
),
|
||||
// Step 2: an identity without an epoch record.
|
||||
(vec![prepare(&instance(INSTANCE))], MindRefusal::MissingIdentity),
|
||||
// Step 4: an epoch record without an identity.
|
||||
(vec![epoch()], MindRefusal::MissingIdentity),
|
||||
// Step 4: two identities are no identity.
|
||||
(
|
||||
vec![epoch(), prepare(&instance(INSTANCE)), prepare(&instance("thought-cage"))],
|
||||
MindRefusal::MissingIdentity,
|
||||
),
|
||||
];
|
||||
for (rows, expected) in cases {
|
||||
let store = planted(rows);
|
||||
let before = store.rows();
|
||||
let refusal = Mind::open_with(store.clone(), &yggdrasil).err().expect("refused");
|
||||
assert_eq!(refusal, expected);
|
||||
assert_eq!(store.pull_count(), 1, "{expected:?}: refused before attaching");
|
||||
assert_eq!(store.rows(), before, "{expected:?}: bytes unchanged");
|
||||
}
|
||||
// A current record beside a foreign one is refused on the count, not
|
||||
// on whichever of the two the store happens to return first.
|
||||
let pair = (epoch(), foreign_epoch());
|
||||
for rows in [vec![pair.0.clone(), pair.1.clone()], vec![pair.1, pair.0]] {
|
||||
assert_eq!(
|
||||
refuse_foreign_epoch(&rows).err(),
|
||||
Some(MindRefusal::ForeignEpoch { found: "2 records".into(), expected: PIPELINE_SCHEMA_EPOCH.into() })
|
||||
);
|
||||
}
|
||||
// A store that passes every gate attaches, which is the second pull.
|
||||
let store = planted(vec![epoch(), prepare(&instance(INSTANCE))]);
|
||||
let mind = Mind::open_with(store.clone(), &yggdrasil).unwrap();
|
||||
assert_eq!(store.pull_count(), 2);
|
||||
assert_eq!(mind.envelopes().len(), 2);
|
||||
assert!(!mind.is_empty());
|
||||
let empty = MemoryStore::new();
|
||||
assert!(Mind::open_with(empty, &yggdrasil).unwrap().is_empty());
|
||||
}
|
||||
|
||||
/// RS-1: receipt v2 is a new type id, so a store carrying an old,
|
||||
/// v1-shaped receipt is refused at the type gate, before anything tries
|
||||
/// to decode its 7-slot payload as the 8-slot v2 shape (F3: with
|
||||
/// `#[serde(default)]` that decode would succeed and silently misread
|
||||
/// every old receipt's ordinal as 0).
|
||||
#[test]
|
||||
fn a_v1_receipt_store_is_refused_at_open() {
|
||||
let store =
|
||||
planted(vec![epoch(), prepare(&instance(INSTANCE)), foreign("huginn.mind_commit_receipt.v1")]);
|
||||
assert_eq!(
|
||||
Mind::open_with(store, &slug(INSTANCE)).err(),
|
||||
Some(MindRefusal::ForeignStore { r#type: "huginn.mind_commit_receipt.v1".into() })
|
||||
);
|
||||
}
|
||||
|
||||
/// RS-1, F5: a real epoch bump always moves every type id (F4), so a
|
||||
/// store written at a foreign epoch also carries at least one foreign
|
||||
/// type. The epoch gate must run first, or the type gate fires instead
|
||||
/// and the epoch gate is never reached for the one case it exists for.
|
||||
/// The eight cases of the opener test above are unchanged by the swap.
|
||||
#[test]
|
||||
fn a_store_written_at_the_previous_epoch_is_refused_by_the_epoch_gate() {
|
||||
let raw = vec![foreign_epoch(), foreign("epiphany.pipeline.campaign.v0")];
|
||||
assert_eq!(
|
||||
Mind::open_with(planted(raw), &slug(INSTANCE)).err(),
|
||||
Some(MindRefusal::ForeignEpoch { found: FOREIGN_EPOCH.into(), expected: PIPELINE_SCHEMA_EPOCH.into() })
|
||||
);
|
||||
}
|
||||
|
||||
/// A foreign store with no epoch record at all still refuses
|
||||
/// `MissingIdentity`, the epoch-first order's own answer -- not
|
||||
/// `ForeignStore`, which is what a mutant that only reorders the gates
|
||||
/// when there *is* some epoch record present would answer, by running
|
||||
/// the type gate first over a store that has no epoch record to look at.
|
||||
/// Pinned by this test.
|
||||
#[test]
|
||||
fn a_foreign_type_with_no_epoch_record_is_missing_identity_not_foreign_store() {
|
||||
let store = planted(vec![foreign(NEVER_A_KNOWN_TYPE)]);
|
||||
assert_eq!(Mind::open_with(store, &slug(INSTANCE)).err(), Some(MindRefusal::MissingIdentity));
|
||||
}
|
||||
|
||||
/// F3 (RS fix batch): an epoch record at the current key, but a
|
||||
/// foreign value, beside a foreign type, still refuses `ForeignEpoch` --
|
||||
/// the epoch gate reads the record's *value* before the type gate ever
|
||||
/// runs, not only when the record's *key* is itself foreign. A mutant
|
||||
/// that reorders the gates only on a foreign key would answer
|
||||
/// `ForeignStore` here instead, since this record sits at the correct
|
||||
/// key.
|
||||
#[test]
|
||||
fn a_foreign_epoch_value_at_the_current_key_beside_a_foreign_type_is_foreign_epoch() {
|
||||
let store = planted(vec![epoch_record(PIPELINE_SCHEMA_EPOCH, FOREIGN_EPOCH), foreign(NEVER_A_KNOWN_TYPE)]);
|
||||
assert_eq!(
|
||||
Mind::open_with(store, &slug(INSTANCE)).err(),
|
||||
Some(MindRefusal::ForeignEpoch { found: FOREIGN_EPOCH.into(), expected: PIPELINE_SCHEMA_EPOCH.into() })
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_instance_document_is_the_identity_not_the_path() {
|
||||
// The same bytes opened under another name are refused.
|
||||
let store = planted(vec![epoch(), prepare(&instance(INSTANCE))]);
|
||||
assert_eq!(
|
||||
Mind::open_with(store, &slug("thought-cage")).err(),
|
||||
Some(MindRefusal::ForeignInstance { declared: "thought-cage".into(), mind: INSTANCE.into() })
|
||||
);
|
||||
|
||||
// A real store copied into another instance's directory is still
|
||||
// yggdrasil's mind: it opens as yggdrasil, and not as thought-cage.
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let written = Mind::path_for(root.path(), &slug(INSTANCE));
|
||||
{
|
||||
let mut store = OwnedRedbMessagePackBackingStore::new(&written).unwrap();
|
||||
store.push(&epoch()).unwrap();
|
||||
store.push(&prepare(&instance(INSTANCE))).unwrap();
|
||||
}
|
||||
let moved = Mind::path_for(root.path(), &slug("thought-cage"));
|
||||
std::fs::create_dir_all(moved.parent().unwrap()).unwrap();
|
||||
std::fs::copy(&written, &moved).unwrap();
|
||||
assert_eq!(
|
||||
Mind::open(root.path(), &slug("thought-cage")).err(),
|
||||
Some(MindRefusal::ForeignInstance { declared: "thought-cage".into(), mind: INSTANCE.into() })
|
||||
);
|
||||
let store = OwnedRedbMessagePackBackingStore::new(&moved).unwrap();
|
||||
let mind = Mind::open_with(store, &slug(INSTANCE)).unwrap();
|
||||
assert_eq!(mind.instance(), &slug(INSTANCE));
|
||||
assert_eq!(mind.envelopes().len(), 2);
|
||||
}
|
||||
|
||||
/// One check, one owner: the value admission refuses a foreign instance
|
||||
/// with is the value `require_instance` returns, so the daemon asking it
|
||||
/// for a read and `admit_steps` asking it for a write cannot disagree.
|
||||
///
|
||||
/// The foreign names are chosen so the check cannot pass by resembling a
|
||||
/// comparison. `NEAR_INSTANCE` is the mind's own length and shares all but
|
||||
/// its last byte, so a check reading only the lengths, only the first byte,
|
||||
/// or running only when the declared name is longer lets it through;
|
||||
/// `PREFIXED_INSTANCE` has the mind's whole name as its prefix, so a check
|
||||
/// asking whether one starts with the other lets that one through; and
|
||||
/// `CASED_INSTANCE` differs in case alone, which a slug label permits, so a
|
||||
/// check folding case answers for a mind that is not this one.
|
||||
#[test]
|
||||
fn require_instance_is_the_one_check_admission_and_the_daemon_share() {
|
||||
use crate::fixtures::{
|
||||
CASED_INSTANCE, DOTTED_INSTANCE, NEAR_INSTANCE, OTHER_INSTANCE, PREFIXED_INSTANCE, UNDERSCORE_INSTANCE,
|
||||
now, opened, provenance, seeded,
|
||||
};
|
||||
use crate::receipt::Faculty;
|
||||
|
||||
let mut mind = seeded();
|
||||
let foreign = MindRefusal::ForeignInstance { declared: OTHER_INSTANCE.into(), mind: INSTANCE.into() };
|
||||
assert_eq!(mind.require_instance(&slug(OTHER_INSTANCE)).err(), Some(foreign.clone()));
|
||||
assert_eq!(mind.require_instance(&slug(INSTANCE)), Ok(()));
|
||||
|
||||
assert_eq!(NEAR_INSTANCE.len(), INSTANCE.len(), "the near name is the mind's own length");
|
||||
assert_eq!(NEAR_INSTANCE[..INSTANCE.len() - 1], INSTANCE[..INSTANCE.len() - 1], "and differs in one byte");
|
||||
assert!(PREFIXED_INSTANCE.starts_with(INSTANCE), "the prefixed name carries the mind's whole name");
|
||||
assert_eq!(CASED_INSTANCE.to_ascii_lowercase(), INSTANCE, "the cased name differs in case alone");
|
||||
assert_ne!(CASED_INSTANCE, INSTANCE);
|
||||
for declared in [NEAR_INSTANCE, PREFIXED_INSTANCE, CASED_INSTANCE] {
|
||||
assert_eq!(
|
||||
mind.require_instance(&slug(declared)).err(),
|
||||
Some(MindRefusal::ForeignInstance { declared: declared.into(), mind: INSTANCE.into() }),
|
||||
"{declared} is not {INSTANCE}"
|
||||
);
|
||||
}
|
||||
|
||||
let batch = crate::admission::PipelineAdmissionBatch {
|
||||
instance: slug(OTHER_INSTANCE),
|
||||
provenance: provenance(Faculty::Hands),
|
||||
documents: vec![instance(OTHER_INSTANCE)],
|
||||
};
|
||||
assert_eq!(mind.admit(batch, now()), crate::admission::PipelineAdmissionOutcome::Refused(foreign));
|
||||
|
||||
// `INSTANCE` ("yggdrasil") carries no separator, so no fold of `_` or
|
||||
// `.` into `-` can ever equate a declared name with it; a mind whose
|
||||
// own name carries one is required to tell the fold apart. Opened as
|
||||
// `OTHER_INSTANCE` ("thought-cage"), a declared name that folds to the
|
||||
// same string under such a comparison must still be refused.
|
||||
let separated = opened(MemoryStore::new(), OTHER_INSTANCE);
|
||||
assert_eq!(UNDERSCORE_INSTANCE.replace('_', "-"), OTHER_INSTANCE, "differs from OTHER_INSTANCE by `_` alone");
|
||||
assert_eq!(DOTTED_INSTANCE.replace('.', "-"), OTHER_INSTANCE, "differs from OTHER_INSTANCE by `.` alone");
|
||||
assert_ne!(UNDERSCORE_INSTANCE, OTHER_INSTANCE);
|
||||
assert_ne!(DOTTED_INSTANCE, OTHER_INSTANCE);
|
||||
for declared in [UNDERSCORE_INSTANCE, DOTTED_INSTANCE] {
|
||||
assert_eq!(
|
||||
separated.require_instance(&slug(declared)).err(),
|
||||
Some(MindRefusal::ForeignInstance { declared: declared.into(), mind: OTHER_INSTANCE.into() }),
|
||||
"{declared} is not {OTHER_INSTANCE}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// One generated test in place of more fixture pairs. The base name
|
||||
/// carries all three separator bytes a `Slug` permits -- `_`, `.` and `-`
|
||||
/// -- at three non-adjacent positions, so every combination of swapping
|
||||
/// one separator for another at a fixed position is itself a grammatical,
|
||||
/// distinct `Slug`. The cartesian product over the three positions (26
|
||||
/// variants, the all-unchanged combination excepted) covers every
|
||||
/// same-position permutation, including a fold that unifies two distinct
|
||||
/// separators (`_` and `.` folded to one canonical byte) and a fold
|
||||
/// applied to only one side of the comparison (the mind's own name
|
||||
/// folded, the declared name left as written, or the reverse). Three more
|
||||
/// variants pad the base with a leading dash, a trailing dash, and a
|
||||
/// doubled interior dash -- all still grammatical, since a label's
|
||||
/// hyphens carry no position or run restriction -- to reach `trim_matches`
|
||||
/// and run-collapsing folds a same-position swap cannot produce. Every
|
||||
/// one of the 30 variants must be refused as `ForeignInstance`, never
|
||||
/// silently admitted as the mind's own name.
|
||||
#[test]
|
||||
fn every_separator_variant_of_a_declared_name_is_foreign_not_the_mind() {
|
||||
const BASE: &str = "ab_cd.ef-gh";
|
||||
let separators: Vec<(usize, char)> =
|
||||
BASE.char_indices().filter(|(_, c)| matches!(c, '_' | '.' | '-')).collect();
|
||||
assert_eq!(separators.len(), 3, "the base carries exactly one of each separator");
|
||||
|
||||
let mut variants: Vec<String> = Vec::new();
|
||||
for a in ['_', '.', '-'] {
|
||||
for b in ['_', '.', '-'] {
|
||||
for c in ['_', '.', '-'] {
|
||||
let mut bytes: Vec<char> = BASE.chars().collect();
|
||||
bytes[separators[0].0] = a;
|
||||
bytes[separators[1].0] = b;
|
||||
bytes[separators[2].0] = c;
|
||||
variants.push(bytes.into_iter().collect());
|
||||
}
|
||||
}
|
||||
}
|
||||
variants.retain(|variant| variant != BASE);
|
||||
assert_eq!(variants.len(), 26, "3^3 combinations, less the one that reproduces BASE");
|
||||
|
||||
variants.push(format!("-{BASE}"));
|
||||
variants.push(format!("{BASE}-"));
|
||||
variants.push(format!("-{BASE}-"));
|
||||
variants.push(BASE.replacen('-', "--", 1));
|
||||
|
||||
let mut seen = std::collections::BTreeSet::new();
|
||||
variants.retain(|variant| seen.insert(variant.clone()));
|
||||
assert_eq!(variants.len(), 30, "26 same-length separator permutations plus 4 padded variants, none colliding");
|
||||
|
||||
let mind = crate::fixtures::opened(MemoryStore::new(), BASE);
|
||||
for declared in &variants {
|
||||
assert!(declared.split('.').all(|label| !label.is_empty()), "{declared}: still grammatical");
|
||||
assert_eq!(
|
||||
mind.require_instance(&slug(declared)).err(),
|
||||
Some(MindRefusal::ForeignInstance { declared: declared.clone(), mind: BASE.into() }),
|
||||
"{declared} must be refused as foreign, not folded into {BASE}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_mind_has_one_owner_at_a_time() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let yggdrasil = slug(INSTANCE);
|
||||
let first = Mind::open(root.path(), &yggdrasil).unwrap();
|
||||
assert!(first.is_empty());
|
||||
let path = Mind::path_for(root.path(), &yggdrasil).display().to_string();
|
||||
assert_eq!(
|
||||
Mind::open(root.path(), &yggdrasil).err(),
|
||||
Some(MindRefusal::MindAlreadyOwned { path: path.clone() })
|
||||
);
|
||||
drop(first);
|
||||
let second = Mind::open(root.path(), &yggdrasil).unwrap();
|
||||
assert!(second.is_empty());
|
||||
assert_eq!(Mind::path_for(root.path(), &yggdrasil), root.path().join("minds").join(INSTANCE).join("mind.redb"));
|
||||
}
|
||||
|
||||
/// N1, Soul's probe: `Mind::open(outer/inner, Slug("..\..\escaped"))` used
|
||||
/// to return `Ok` and create `outer/escaped/mind.redb`, a store outside
|
||||
/// the state root nothing beneath `outer/inner` names. `path_for` joins a
|
||||
/// declared name as written and never validated it, so a `Slug` carrying
|
||||
/// `..` and a path separator escaped through the one door that touches the
|
||||
/// filesystem. `require_grammatical_slug` now runs before `path_for` is
|
||||
/// even called, so the store is never created and the refusal is the
|
||||
/// leaf's own grammar, not a filesystem error surfacing later.
|
||||
#[test]
|
||||
fn a_slug_outside_the_grammar_cannot_escape_the_state_root() {
|
||||
let outer = tempfile::tempdir().unwrap();
|
||||
let inner = outer.path().join("inner");
|
||||
std::fs::create_dir_all(&inner).unwrap();
|
||||
|
||||
let escaping = Slug("..\\..\\escaped".into());
|
||||
assert_eq!(
|
||||
Mind::open(&inner, &escaping).err(),
|
||||
Some(MindRefusal::Document(epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "instance".into(),
|
||||
value: escaping.0.clone(),
|
||||
}))
|
||||
);
|
||||
|
||||
assert!(!outer.path().join("escaped").join("mind.redb").exists(), "no store escaped the state root");
|
||||
assert!(!inner.join("minds").exists(), "no store was created under the state root either");
|
||||
}
|
||||
|
||||
/// `open_with` calls `open_checked`, the one construction site `open`
|
||||
/// shares with every store already in hand (planted directly, or opened
|
||||
/// by a test through `store_path_for`), so it must run
|
||||
/// `require_grammatical_slug` too, not only `open`'s own path-joining
|
||||
/// caller. A store need not hold anything for this to refuse: the
|
||||
/// declared name is checked before the store is even pulled.
|
||||
#[test]
|
||||
fn open_with_refuses_a_declared_instance_outside_the_grammar() {
|
||||
let store = MemoryStore::new();
|
||||
let bad = Slug("not a slug!".into());
|
||||
assert_eq!(
|
||||
Mind::open_with(store.clone(), &bad).err(),
|
||||
Some(MindRefusal::Document(epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "instance".into(),
|
||||
value: bad.0.clone(),
|
||||
}))
|
||||
);
|
||||
assert_eq!(store.pull_count(), 0, "refused before the store was ever pulled");
|
||||
}
|
||||
|
||||
/// `require_instance` is A1 for admission and the daemon's reads
|
||||
/// alike, and it runs the same grammar door on the *declared* name before
|
||||
/// comparing it to the mind's own. A declared name outside the grammar is
|
||||
/// `InvalidFormat { field: "declared" }`, never folded into a comparison
|
||||
/// or silently treated as merely foreign.
|
||||
#[test]
|
||||
fn require_instance_refuses_a_declared_name_outside_the_grammar() {
|
||||
let mind = crate::fixtures::seeded();
|
||||
let bad = Slug("not a slug!".into());
|
||||
assert_eq!(
|
||||
mind.require_instance(&bad).err(),
|
||||
Some(MindRefusal::Document(epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "declared".into(),
|
||||
value: bad.0.clone(),
|
||||
}))
|
||||
);
|
||||
}
|
||||
|
||||
/// A Unicode hyphen (U+2010) is not ASCII `-`, so a grammatical
|
||||
/// `Slug` never contains one and a name carrying it is refused by the
|
||||
/// leaf's own grammar as written -- nothing here may fold it to plain
|
||||
/// `-` before asking `Slug::validate_slug`, the way a fullwidth mutant
|
||||
/// pinned by
|
||||
/// `daemon::tests::a_declared_instance_outside_the_grammar_is_refused_by_name_before_the_identity_check`
|
||||
/// folds a fullwidth letter to its ASCII form.
|
||||
#[test]
|
||||
fn a_unicode_hyphen_is_not_folded_before_the_grammar_check() {
|
||||
let declared = Slug(format!("ab{}cd", '\u{2010}'));
|
||||
assert!(!declared.0.is_ascii(), "U+2010 is not ASCII, unlike plain '-'");
|
||||
assert_eq!(
|
||||
Mind::open_with(MemoryStore::new(), &declared).err(),
|
||||
Some(MindRefusal::Document(epiphany_pipeline::PipelineRefusal::InvalidFormat {
|
||||
field: "instance".into(),
|
||||
value: declared.0.clone(),
|
||||
}))
|
||||
);
|
||||
}
|
||||
}
|
||||
1760
crates/huginn-mind/src/query.rs
Normal file
1760
crates/huginn-mind/src/query.rs
Normal file
File diff suppressed because it is too large
Load diff
438
crates/huginn-mind/src/receipt.rs
Normal file
438
crates/huginn-mind/src/receipt.rs
Normal file
|
|
@ -0,0 +1,438 @@
|
|||
//! The commit receipt: what a batch read and what it wrote, byte for byte.
|
||||
//!
|
||||
//! This is the bounded duplicate of Epiphany's Mind receipt (FU-3), smaller
|
||||
//! by design: no authority enum, no companion documents, no `invariant_owner`,
|
||||
//! no `store_id`. Provenance is a field on the receipt, not a companion, and
|
||||
//! the digest excludes it, so an exact replay across sessions and faculties
|
||||
//! is idempotent. If a third consumer appears, extract the primitive into
|
||||
//! CultLib rather than copy it again.
|
||||
//!
|
||||
//! Digest: `receipt_id = "mind-commit-" + hex(sha256(msgpack_named((instance,
|
||||
//! strong_reads, writes))))`, where each list is `DocumentVersion`s in
|
||||
//! `(document_type, document_key)` order and a version carries the exact
|
||||
//! payload bytes and their SHA-256. `committed_at`, `provenance` and
|
||||
//! `ordinal` are not digested: an exact replay keeps the ordinal its first
|
||||
//! admission assigned.
|
||||
|
||||
use chrono::{DateTime, SecondsFormat, Utc};
|
||||
use cultcache_rs::{CultCacheEnvelope, DatabaseEntry};
|
||||
use epiphany_pipeline::{PipelineKind, PipelineRef, Short, Slug};
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::mind::{Mind, unavailable};
|
||||
use crate::refusal::MindRefusal;
|
||||
use crate::store::MindStore;
|
||||
|
||||
pub const RECEIPT_SCHEMA_VERSION: &str = "huginn.mind_commit_receipt.v2";
|
||||
|
||||
/// Which faculty a batch was admitted as. Attribution, not authority
|
||||
/// (ruling 18): no admission rule trusts it, and Cut 9's views read it back
|
||||
/// from the receipt.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub enum Faculty {
|
||||
SelfFaculty,
|
||||
Imagination,
|
||||
Hands,
|
||||
Soul,
|
||||
MindSteward,
|
||||
Eyes,
|
||||
Operator,
|
||||
}
|
||||
|
||||
/// Who admitted a batch: the faculty, the agent, its session and the tool.
|
||||
/// A field on the receipt and nothing else; the digest excludes it.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct PipelineProvenance {
|
||||
pub faculty: Faculty,
|
||||
pub agent: epiphany_pipeline::Short,
|
||||
pub session: epiphany_pipeline::Short,
|
||||
pub tool: epiphany_pipeline::Short,
|
||||
}
|
||||
|
||||
/// The exact bytes of one stored document, as read or as written.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct DocumentVersion {
|
||||
pub document_type: String,
|
||||
pub document_key: String,
|
||||
pub schema_id: Option<String>,
|
||||
pub payload_msgpack: Vec<u8>,
|
||||
pub payload_sha256: String,
|
||||
}
|
||||
|
||||
impl DocumentVersion {
|
||||
pub fn from_envelope(envelope: &CultCacheEnvelope) -> Self {
|
||||
Self {
|
||||
document_type: envelope.r#type.clone(),
|
||||
document_key: envelope.key.clone(),
|
||||
schema_id: envelope.schema_id.clone(),
|
||||
payload_msgpack: envelope.payload.clone(),
|
||||
payload_sha256: sha256_hex(&envelope.payload),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn identity(&self) -> (&str, &str) {
|
||||
(&self.document_type, &self.document_key)
|
||||
}
|
||||
|
||||
fn validate(&self) -> Result<(), MindRefusal> {
|
||||
if self.document_type.is_empty() || self.document_key.is_empty() {
|
||||
return Err(integrity("a document version names no identity"));
|
||||
}
|
||||
if self.payload_sha256 != sha256_hex(&self.payload_msgpack) {
|
||||
return Err(integrity(&format!(
|
||||
"document {}/{} payload digest mismatch",
|
||||
self.document_type, self.document_key
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// The receipt of one admitted batch, stored beside the documents it names
|
||||
/// and keyed by its id.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
|
||||
#[cultcache(type = "huginn.mind_commit_receipt.v2", schema = "HuginnCommitReceipt")]
|
||||
pub struct HuginnCommitReceipt {
|
||||
#[cultcache(key = 0)]
|
||||
pub schema_version: String,
|
||||
#[cultcache(key = 1)]
|
||||
pub receipt_id: String,
|
||||
#[cultcache(key = 2)]
|
||||
pub instance: String,
|
||||
#[cultcache(key = 3)]
|
||||
pub provenance: PipelineProvenance,
|
||||
#[cultcache(key = 4)]
|
||||
pub strong_reads: Vec<DocumentVersion>,
|
||||
#[cultcache(key = 5)]
|
||||
pub writes: Vec<DocumentVersion>,
|
||||
#[cultcache(key = 6)]
|
||||
pub committed_at: String,
|
||||
/// This mind's admission order: dense from 1, never reused, never
|
||||
/// digested. `head` reads it back; `candidate` is the one place it is
|
||||
/// assigned, as `head + 1`.
|
||||
#[cultcache(key = 7)]
|
||||
pub ordinal: u64,
|
||||
}
|
||||
|
||||
impl HuginnCommitReceipt {
|
||||
/// The digest, from what was read and written and never from who wrote
|
||||
/// it or when.
|
||||
pub fn digest(&self) -> Result<String, MindRefusal> {
|
||||
let bytes = rmp_serde::to_vec_named(&(&self.instance, &self.strong_reads, &self.writes))
|
||||
.map_err(|error| integrity(&format!("receipt digest input does not encode: {error}")))?;
|
||||
Ok(format!("mind-commit-{}", sha256_hex(&bytes)))
|
||||
}
|
||||
|
||||
pub fn validate(&self) -> Result<(), MindRefusal> {
|
||||
if self.schema_version != RECEIPT_SCHEMA_VERSION {
|
||||
return Err(integrity(&format!("receipt schema version {:?} is foreign", self.schema_version)));
|
||||
}
|
||||
if self.instance.is_empty() {
|
||||
return Err(integrity("receipt names no instance"));
|
||||
}
|
||||
if self.writes.is_empty() {
|
||||
return Err(integrity("receipt writes nothing"));
|
||||
}
|
||||
for version in self.strong_reads.iter().chain(&self.writes) {
|
||||
version.validate()?;
|
||||
}
|
||||
if !unique_identities(&self.strong_reads) || !unique_identities(&self.writes) {
|
||||
return Err(integrity("receipt names an identity twice"));
|
||||
}
|
||||
if self.ordinal == 0 {
|
||||
return Err(integrity("receipt ordinal is 0"));
|
||||
}
|
||||
if self.receipt_id != self.digest()? {
|
||||
return Err(integrity(&format!("receipt {} does not match its digest", self.receipt_id)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// How a commit ended. An exact replay is decided by admission before the
|
||||
/// commit is attempted, so it is not an exit of the primitive.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum CommitOutcome {
|
||||
/// The batch and its receipt landed whole.
|
||||
Committed(HuginnCommitReceipt),
|
||||
/// The swap lost: a strong read changed underneath, or a write's identity
|
||||
/// already existed. Named where the kind is a pipeline kind.
|
||||
Conflict(Vec<PipelineRef>),
|
||||
}
|
||||
|
||||
/// The receipt a batch would land with: its id is the digest of the exact
|
||||
/// bytes read and written, computed before anything is stored so admission
|
||||
/// can look up an exact replay. This is the one construction site of a
|
||||
/// receipt.
|
||||
pub(crate) fn candidate(
|
||||
instance: &Slug,
|
||||
provenance: PipelineProvenance,
|
||||
strong_reads: &[CultCacheEnvelope],
|
||||
writes: &[CultCacheEnvelope],
|
||||
ordinal: u64,
|
||||
now: DateTime<Utc>,
|
||||
) -> Result<HuginnCommitReceipt, MindRefusal> {
|
||||
let mut receipt = HuginnCommitReceipt {
|
||||
schema_version: RECEIPT_SCHEMA_VERSION.to_string(),
|
||||
receipt_id: String::new(),
|
||||
instance: instance.0.clone(),
|
||||
provenance,
|
||||
strong_reads: versions(strong_reads),
|
||||
writes: versions(writes),
|
||||
committed_at: now.to_rfc3339_opts(SecondsFormat::Secs, true),
|
||||
ordinal,
|
||||
};
|
||||
receipt.receipt_id = receipt.digest()?;
|
||||
Ok(receipt)
|
||||
}
|
||||
|
||||
/// This mind's admission order: `N`, the count of receipts, when their
|
||||
/// ordinals are exactly `{1..=N}`. Every reader of the ordinal asks here,
|
||||
/// never by counting receipts itself: admission takes `head(self)? + 1` for
|
||||
/// the next one, and `AdmissionIndex::build` (`query.rs`) calls it too, so
|
||||
/// `query` and `view` carry the same density check rather than trusting a
|
||||
/// stored ordinal unchecked. `Mind::receipts` is `pub(crate)`, so this crate
|
||||
/// is the only place raw ordinals can be read at all; that keeps this the
|
||||
/// one door, rather than something a doc comment merely asks callers to
|
||||
/// respect. `ordinal` stays display-only on the read side until RS-3 makes
|
||||
/// it the order. A chain that is not dense refuses rather than guessing, on
|
||||
/// both sides alike.
|
||||
pub(crate) fn head<S: MindStore>(mind: &Mind<S>) -> Result<u64, MindRefusal> {
|
||||
let mut ordinals = mind.receipts()?.into_iter().map(|receipt| receipt.ordinal).collect::<Vec<_>>();
|
||||
ordinals.sort_unstable();
|
||||
let n = ordinals.len() as u64;
|
||||
if ordinals.into_iter().eq(1..=n) {
|
||||
Ok(n)
|
||||
} else {
|
||||
Err(MindRefusal::Unavailable { detail: "receipt ordinals are not 1..=N: found a duplicate or a gap".into() })
|
||||
}
|
||||
}
|
||||
|
||||
/// The versions of a set of envelopes in identity order, so the digest does
|
||||
/// not depend on the order a batch was assembled in.
|
||||
fn versions(envelopes: &[CultCacheEnvelope]) -> Vec<DocumentVersion> {
|
||||
let mut versions = envelopes.iter().map(DocumentVersion::from_envelope).collect::<Vec<_>>();
|
||||
versions.sort_by(|left, right| left.identity().cmp(&right.identity()));
|
||||
versions
|
||||
}
|
||||
|
||||
/// The stored receipt with the candidate's id, if the batch already landed:
|
||||
/// its reads and writes must equal the candidate's, or the store holds a
|
||||
/// receipt this binary cannot vouch for.
|
||||
pub(crate) fn replay<S: MindStore>(
|
||||
mind: &Mind<S>,
|
||||
candidate: &HuginnCommitReceipt,
|
||||
) -> Result<Option<HuginnCommitReceipt>, MindRefusal> {
|
||||
let Some(envelope) = mind.raw_envelope(HuginnCommitReceipt::TYPE, &candidate.receipt_id) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let existing: HuginnCommitReceipt = rmp_serde::from_slice(&envelope.payload)
|
||||
.map_err(|error| integrity(&format!("receipt {} does not decode: {error}", candidate.receipt_id)))?;
|
||||
existing.validate()?;
|
||||
if existing.strong_reads != candidate.strong_reads || existing.writes != candidate.writes {
|
||||
return Err(integrity(&format!("receipt {} is stored with other content", candidate.receipt_id)));
|
||||
}
|
||||
Ok(Some(existing))
|
||||
}
|
||||
|
||||
/// Lands a batch whole, or not at all: one compare-and-swap with the strong
|
||||
/// reads as its expectation and the writes plus the receipt as its
|
||||
/// replacements. The image is re-pulled either way; a lost swap is diffed
|
||||
/// against the fresh image and typed `Conflict`.
|
||||
pub(crate) fn commit<S: MindStore>(
|
||||
mind: &mut Mind<S>,
|
||||
receipt: HuginnCommitReceipt,
|
||||
strong_reads: Vec<CultCacheEnvelope>,
|
||||
writes: Vec<CultCacheEnvelope>,
|
||||
) -> Result<CommitOutcome, MindRefusal> {
|
||||
receipt.validate()?;
|
||||
let receipt_envelope = mind
|
||||
.cache()
|
||||
.prepare_entry_named(&receipt.receipt_id, &receipt)
|
||||
.map(|(envelope, _)| envelope)
|
||||
.map_err(unavailable)?;
|
||||
// The store requires every expected identity to be replaced, so each
|
||||
// strong read is re-inserted unchanged beside the writes.
|
||||
let mut replacements = writes.clone();
|
||||
replacements.extend(strong_reads.iter().cloned());
|
||||
replacements.push(receipt_envelope);
|
||||
let expected: &[CultCacheEnvelope] = &strong_reads;
|
||||
let landed = MindStore::compare_and_swap_batch(mind.store(), expected, replacements).map_err(unavailable)?;
|
||||
mind.refresh()?;
|
||||
if !landed {
|
||||
let changed = strong_reads
|
||||
.iter()
|
||||
.filter(|read| mind.raw_envelope(&read.r#type, &read.key) != Some(read))
|
||||
.chain(writes.iter().filter(|write| mind.raw_envelope(&write.r#type, &write.key).is_some()));
|
||||
let identities = changed
|
||||
.filter_map(|envelope| {
|
||||
let kind = PipelineKind::ALL.iter().find(|kind| kind.type_id() == envelope.r#type)?;
|
||||
Some(PipelineRef { kind: *kind, id: Short(envelope.key.clone()) })
|
||||
})
|
||||
.collect();
|
||||
return Ok(CommitOutcome::Conflict(identities));
|
||||
}
|
||||
Ok(CommitOutcome::Committed(receipt))
|
||||
}
|
||||
|
||||
fn unique_identities(versions: &[DocumentVersion]) -> bool {
|
||||
let identities = versions.iter().map(DocumentVersion::identity).collect::<std::collections::BTreeSet<_>>();
|
||||
identities.len() == versions.len()
|
||||
}
|
||||
|
||||
fn integrity(detail: &str) -> MindRefusal {
|
||||
MindRefusal::Unavailable { detail: detail.into() }
|
||||
}
|
||||
|
||||
pub(crate) fn sha256_hex(bytes: &[u8]) -> String {
|
||||
format!("{:x}", Sha256::digest(bytes))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use super::*;
|
||||
use crate::fixtures::{INSTANCE, admit, committed, id, instance, now, opened, prepare, provenance, question, refusal, seed, slug};
|
||||
use crate::mind::schema_cache;
|
||||
use crate::store::test_stores::MemoryStore;
|
||||
use cultcache_rs::{CacheBackingStore, PushAllOptions};
|
||||
use epiphany_pipeline::{PipelineDocument, PipelineKind, PipelineRefusal};
|
||||
|
||||
/// A memory store that keeps the replacement set of every swap it was
|
||||
/// asked for, so a test can count the commit's calls as well as read
|
||||
/// their contents.
|
||||
#[derive(Clone)]
|
||||
struct RecordingStore {
|
||||
inner: MemoryStore,
|
||||
calls: Arc<Mutex<Vec<Vec<CultCacheEnvelope>>>>,
|
||||
}
|
||||
|
||||
impl RecordingStore {
|
||||
fn new() -> Self {
|
||||
Self { inner: MemoryStore::new(), calls: Arc::new(Mutex::new(Vec::new())) }
|
||||
}
|
||||
|
||||
fn forget(&self) {
|
||||
self.calls.lock().unwrap().clear();
|
||||
}
|
||||
|
||||
fn calls(&self) -> Vec<Vec<CultCacheEnvelope>> {
|
||||
self.calls.lock().unwrap().clone()
|
||||
}
|
||||
}
|
||||
|
||||
impl CacheBackingStore for RecordingStore {
|
||||
fn pull_all(&self) -> anyhow::Result<Vec<CultCacheEnvelope>> {
|
||||
self.inner.pull_all()
|
||||
}
|
||||
|
||||
fn push(&mut self, entry: &CultCacheEnvelope) -> anyhow::Result<()> {
|
||||
self.inner.push(entry)
|
||||
}
|
||||
|
||||
fn delete(&mut self, entry: &CultCacheEnvelope) -> anyhow::Result<()> {
|
||||
self.inner.delete(entry)
|
||||
}
|
||||
|
||||
fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> anyhow::Result<()> {
|
||||
self.inner.push_all(entries, options)
|
||||
}
|
||||
}
|
||||
|
||||
impl MindStore for RecordingStore {
|
||||
fn compare_and_swap_batch(
|
||||
&self,
|
||||
expected: &[CultCacheEnvelope],
|
||||
replacements: Vec<CultCacheEnvelope>,
|
||||
) -> anyhow::Result<bool> {
|
||||
self.calls.lock().unwrap().push(replacements.clone());
|
||||
self.inner.compare_and_swap_batch(expected, replacements)
|
||||
}
|
||||
}
|
||||
|
||||
/// The receipt is part of the batch's own swap, never a second one after
|
||||
/// it: a receipt written separately could land while the documents did
|
||||
/// not, or the reverse.
|
||||
#[test]
|
||||
fn a_commit_is_one_swap_carrying_the_documents_and_the_receipt_together() {
|
||||
let store = RecordingStore::new();
|
||||
let mut mind = opened(store.clone(), INSTANCE);
|
||||
seed(&mut mind);
|
||||
store.forget();
|
||||
let (receipt_id, _) = committed(admit(&mut mind, vec![question("Q1", &["A", "B"], "A")]));
|
||||
let calls = store.calls();
|
||||
assert_eq!(calls.len(), 1, "one commit, one swap");
|
||||
let landed = calls[0].iter().map(|entry| (entry.r#type.clone(), entry.key.clone())).collect::<Vec<_>>();
|
||||
assert_eq!(landed.len(), 2, "{landed:?}");
|
||||
assert!(landed.contains(&(PipelineKind::Question.type_id().to_string(), id("question", "Q1"))), "{landed:?}");
|
||||
assert!(landed.contains(&(HuginnCommitReceipt::TYPE.to_string(), receipt_id)), "{landed:?}");
|
||||
}
|
||||
|
||||
/// RS-1: `head` requires the stored ordinals to be exactly `{1..=N}`.
|
||||
/// A duplicate and a gap are planted directly, bypassing the commit path,
|
||||
/// and both `head` itself and a fresh admission over the same store must
|
||||
/// refuse. `query.rs`'s
|
||||
/// `query_and_view_refuse_a_receipt_chain_that_is_not_dense` pins the
|
||||
/// same check on the read side.
|
||||
#[test]
|
||||
fn a_chain_that_is_not_dense_refuses_head_and_admission() {
|
||||
let base = MemoryStore::new();
|
||||
let mut mind = opened(base.clone(), INSTANCE);
|
||||
seed(&mut mind);
|
||||
committed(admit(&mut mind, vec![question("Q1", &["A", "B"], "A")]));
|
||||
assert_eq!(head(&mind).unwrap(), 2, "the seed and Q1's batch");
|
||||
|
||||
let cache = schema_cache().unwrap();
|
||||
let plant_at = |ordinal: u64, label: &str| {
|
||||
let broken = MemoryStore::new();
|
||||
for row in base.rows() {
|
||||
broken.plant(row);
|
||||
}
|
||||
let extra = candidate(
|
||||
&slug(INSTANCE),
|
||||
provenance(Faculty::Hands),
|
||||
&[],
|
||||
&[prepare(&question(label, &["A", "B"], "A"))],
|
||||
ordinal,
|
||||
now(),
|
||||
)
|
||||
.unwrap();
|
||||
broken.plant(cache.prepare_entry_named(&extra.receipt_id, &extra).unwrap().0);
|
||||
broken
|
||||
};
|
||||
|
||||
// A duplicate: a third receipt at ordinal 2, which the store already
|
||||
// carries.
|
||||
let duplicated = plant_at(2, "Q2");
|
||||
assert!(matches!(head(&opened(duplicated.clone(), INSTANCE)), Err(MindRefusal::Unavailable { .. })));
|
||||
let mut duplicated_mind = opened(duplicated, INSTANCE);
|
||||
assert!(matches!(
|
||||
refusal(admit(&mut duplicated_mind, vec![question("Q3", &["A", "B"], "A")])),
|
||||
MindRefusal::Unavailable { .. }
|
||||
));
|
||||
|
||||
// A gap: a third receipt at ordinal 4, skipping 3.
|
||||
let gapped = plant_at(4, "Q2");
|
||||
assert!(matches!(head(&opened(gapped.clone(), INSTANCE)), Err(MindRefusal::Unavailable { .. })));
|
||||
let mut gapped_mind = opened(gapped, INSTANCE);
|
||||
assert!(matches!(
|
||||
refusal(admit(&mut gapped_mind, vec![question("Q3", &["A", "B"], "A")])),
|
||||
MindRefusal::Unavailable { .. }
|
||||
));
|
||||
}
|
||||
|
||||
/// S5 closed against the real type: the organ's own receipt in a mind's
|
||||
/// store is never decoded as a pipeline document.
|
||||
#[test]
|
||||
fn decode_refuses_the_organs_own_receipt() {
|
||||
let mut envelope = prepare(&instance(INSTANCE));
|
||||
envelope.r#type = HuginnCommitReceipt::TYPE.into();
|
||||
assert_eq!(
|
||||
PipelineDocument::decode(&envelope),
|
||||
Err(PipelineRefusal::ForeignStore { r#type: "huginn.mind_commit_receipt.v2".into() })
|
||||
);
|
||||
}
|
||||
}
|
||||
147
crates/huginn-mind/src/refusal.rs
Normal file
147
crates/huginn-mind/src/refusal.rs
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
//! Typed refusals of the organ. A refusal is data with a field an agent can
|
||||
//! act on, never a transport error: the outcome carries it, the wire (Cut 10)
|
||||
//! serialises it, the tools (Cut 13) show it.
|
||||
|
||||
use cultnet_rs::SelectionRefusal;
|
||||
use epiphany_pipeline::{PipelineKind, PipelineRefusal};
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Every way a mind refuses. `Document` wraps the leaf's four (bounds,
|
||||
/// formats, key identity, foreign type); the rest are the organ's: store
|
||||
/// identity, epoch, ownership, batch shape, references, and every cross-field
|
||||
/// and cross-document rule of admission.
|
||||
///
|
||||
/// The two sequenced kinds refuse in the same pair of shapes: an
|
||||
/// `OutOfSequence` when the writer's `sequence` is not the previous plus one,
|
||||
/// and `AlreadyResolved`/`AlreadyStewarded` when a record of that scope is
|
||||
/// still in force.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub enum MindRefusal {
|
||||
Document(PipelineRefusal),
|
||||
ForeignInstance { declared: String, mind: String },
|
||||
MissingIdentity,
|
||||
ForeignEpoch { found: String, expected: String },
|
||||
ForeignStore { r#type: String },
|
||||
MindAlreadyOwned { path: String },
|
||||
BatchSize { actual: u32 },
|
||||
IdentityCollision { kind: PipelineKind, id: String },
|
||||
MissingReference { kind: PipelineKind, id: String },
|
||||
AlreadyResolved { subject: String },
|
||||
ResolutionOutOfSequence { subject: String, expected: u32, actual: u32 },
|
||||
AlreadyStewarded { repo: String },
|
||||
StewardshipOutOfSequence { repo: String, expected: u32, actual: u32 },
|
||||
IncompatibleResolution { subject_kind: PipelineKind, outcome: String },
|
||||
WouldReinstateOverLater { subject: String, later: String },
|
||||
CitesResolvedDocument { kind: PipelineKind, id: String },
|
||||
EmptySupersession,
|
||||
UnknownSupersessor { id: String },
|
||||
RevisionWithoutSupersession { kind: PipelineKind, revision: u32 },
|
||||
DuplicateLabel { field: String, label: String },
|
||||
InvalidOptions { question: String },
|
||||
InvalidChoice { ruling: String, choice: String },
|
||||
QuoteWithoutOperator { ruling: String },
|
||||
EmptyRepos { campaign: String },
|
||||
RepoNotStewarded { repo: String },
|
||||
RepoNotInCampaign { repo: String },
|
||||
CutReportWithoutSpec { report: String },
|
||||
SpecMismatch { field: String },
|
||||
RangeOutsideCommits { head: String },
|
||||
FalsifiedClaimWithoutConfirmedFinding { claim: String },
|
||||
UnprovenClaimWithConfirmedFinding { claim: String },
|
||||
PromiseWithoutVerdict { promise: String },
|
||||
UnknownMutationLabel { label: String },
|
||||
FindingWithoutRange,
|
||||
FindingWithoutEvidence,
|
||||
UnknownInvariant { label: String },
|
||||
NotStewarded { repo: String },
|
||||
/// The one refusal a mind never raises: the answer exceeds the largest
|
||||
/// body the organ will deliver by any plane, one send or a deferred body.
|
||||
/// It lives here because a refusal rides the response schema and there is
|
||||
/// one of those, so a client reads it the way it reads every other refusal
|
||||
/// rather than parsing a second vocabulary. `bytes` is the encoded answer
|
||||
/// (the payload, not its envelope) and `limit` is the largest body the
|
||||
/// daemon defers, both so a caller can narrow its own request; nothing is
|
||||
/// truncated or paginated on its behalf. The daemon owns it and its
|
||||
/// deferral bound decides it; no rule of the mind is involved.
|
||||
ResponseTooLarge { bytes: u64, limit: u64 },
|
||||
/// A selection the substrate or the organ's value door refused: `field`
|
||||
/// is the selection field the client sent, `value` the offending part.
|
||||
SelectionInvalid { field: String, value: Option<String>, message: String },
|
||||
/// A cursor that does not decode, does not belong to this selection or
|
||||
/// was not minted by this process, or names a snapshot this mind has not
|
||||
/// reached.
|
||||
CursorInvalid { message: String },
|
||||
Unavailable { detail: String },
|
||||
}
|
||||
|
||||
impl std::fmt::Display for MindRefusal {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(formatter, "mind refusal: {self:?}")
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for MindRefusal {}
|
||||
|
||||
impl From<PipelineRefusal> for MindRefusal {
|
||||
fn from(refusal: PipelineRefusal) -> Self {
|
||||
Self::Document(refusal)
|
||||
}
|
||||
}
|
||||
|
||||
/// The one total mapping from the substrate's refusals, with no wildcard arm.
|
||||
/// `CursorStale` is unreachable: the organ answers at the cursor's own `asOf`,
|
||||
/// so a stale refusal is an organ defect and says so. `ReferenceOutsideTarget`
|
||||
/// is integrity: A7 admits only referents of the declared kind, so a stored
|
||||
/// edge outside its target means the store is not what admission wrote.
|
||||
impl From<SelectionRefusal> for MindRefusal {
|
||||
fn from(refusal: SelectionRefusal) -> Self {
|
||||
match refusal {
|
||||
SelectionRefusal::Invalid(invalid) => {
|
||||
Self::SelectionInvalid { field: invalid.field, value: invalid.value, message: invalid.message }
|
||||
}
|
||||
SelectionRefusal::CursorInvalid { message } => Self::CursorInvalid { message },
|
||||
SelectionRefusal::CursorStale { as_of, current } => Self::Unavailable {
|
||||
detail: format!("cursor_stale: the organ answers at the cursor's asOf ({as_of}), yet was asked at {current}"),
|
||||
},
|
||||
reference @ SelectionRefusal::ReferenceOutsideTarget { .. } => {
|
||||
Self::Unavailable { detail: format!("stored reference outside its declared target: {reference}") }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use cultnet_rs::SelectionInvalid;
|
||||
|
||||
/// The mapping is the organ's word on each substrate refusal: a bad
|
||||
/// selection and a bad cursor cross as themselves, and the two the organ
|
||||
/// is built never to raise cross as integrity faults, not as anything a
|
||||
/// client could act on.
|
||||
#[test]
|
||||
fn each_substrate_refusal_maps_to_one_organ_refusal() {
|
||||
let invalid = SelectionInvalid { field: "keys".into(), value: Some("x".into()), message: "no".into() };
|
||||
assert_eq!(
|
||||
MindRefusal::from(SelectionRefusal::Invalid(invalid)),
|
||||
MindRefusal::SelectionInvalid { field: "keys".into(), value: Some("x".into()), message: "no".into() }
|
||||
);
|
||||
assert_eq!(
|
||||
MindRefusal::from(SelectionRefusal::CursorInvalid { message: "bad".into() }),
|
||||
MindRefusal::CursorInvalid { message: "bad".into() }
|
||||
);
|
||||
assert!(matches!(
|
||||
MindRefusal::from(SelectionRefusal::CursorStale { as_of: 3, current: 4 }),
|
||||
MindRefusal::Unavailable { detail } if detail.contains("cursor_stale")
|
||||
));
|
||||
let outside = SelectionRefusal::ReferenceOutsideTarget {
|
||||
from_schema_id: "a".into(),
|
||||
from_key: "k".into(),
|
||||
role: "answers".into(),
|
||||
to_schema_id: "b".into(),
|
||||
to_key: "j".into(),
|
||||
};
|
||||
assert!(matches!(MindRefusal::from(outside), MindRefusal::Unavailable { .. }));
|
||||
}
|
||||
}
|
||||
412
crates/huginn-mind/src/rows.rs
Normal file
412
crates/huginn-mind/src/rows.rs
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
//! What the read side gives CultNet's selection evaluator: the rows, the
|
||||
//! organ's closed vocabulary, and the door that checks the values a selection
|
||||
//! carries. Types, rules and boundaries only.
|
||||
//!
|
||||
//! The substrate owns the grammar of a selection's names, the order, the
|
||||
//! cursor, the hop and paging. This module owns what a row answers for each
|
||||
//! declared alias, which aliases and roles exist, and the value domain of
|
||||
//! every alias: `cultnet_rs::validate` refuses a name it does not know and
|
||||
//! never looks at a value, so a malformed filter would read as an empty
|
||||
//! answer. `refuse_values` is the door that closes that (S6).
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use cultnet_rs::{Row, RowSet, Selection};
|
||||
use epiphany_pipeline::{OrgRepo, Label, PipelineDocument, PipelineKind, PipelineRef, ResolutionOutcome, Slug};
|
||||
|
||||
use crate::docs::{CitationRole, Held, kind_of_id, kind_of_type};
|
||||
use crate::query::PipelineDocumentView;
|
||||
use crate::refusal::MindRefusal;
|
||||
|
||||
/// The declared index aliases: what a row answers `values(alias)` with.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Alias {
|
||||
/// The key's root segment: a campaign slug, or an instance for
|
||||
/// stewardships, hand-offs and the identity document. A resolution's root
|
||||
/// is its subject's.
|
||||
Root,
|
||||
InForce,
|
||||
Faculty,
|
||||
/// `OrgRepo` identity spelling (lowercase), read through `base`.
|
||||
Repo,
|
||||
/// The label in `cut-<label>.`, read through `base`.
|
||||
Cut,
|
||||
Severity,
|
||||
Confidence,
|
||||
Origin,
|
||||
Authority,
|
||||
ClaimOutcome,
|
||||
Outcome,
|
||||
}
|
||||
|
||||
const FACULTIES: [&str; 7] = ["SelfFaculty", "Imagination", "Hands", "Soul", "MindSteward", "Eyes", "Operator"];
|
||||
const SEVERITIES: [&str; 4] = ["Blocker", "High", "Medium", "Low"];
|
||||
const CONFIDENCES: [&str; 2] = ["Confirmed", "Plausible"];
|
||||
const ORIGINS: [&str; 2] = ["Introduced", "PreExisting"];
|
||||
const AUTHORITIES: [&str; 3] = ["Operator", "Standing", "Defaulted"];
|
||||
const CLAIM_OUTCOMES: [&str; 3] = ["Holds", "Falsified", "Unproven"];
|
||||
const OUTCOMES: [&str; 6] = ["Superseded", "Answered", "Fixed", "Deferred", "Recorded", "Withdrawn"];
|
||||
const BOOLEANS: [&str; 2] = ["true", "false"];
|
||||
|
||||
impl Alias {
|
||||
const ALL: [Alias; 11] = [
|
||||
Self::Root,
|
||||
Self::InForce,
|
||||
Self::Faculty,
|
||||
Self::Repo,
|
||||
Self::Cut,
|
||||
Self::Severity,
|
||||
Self::Confidence,
|
||||
Self::Origin,
|
||||
Self::Authority,
|
||||
Self::ClaimOutcome,
|
||||
Self::Outcome,
|
||||
];
|
||||
|
||||
fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::Root => "root",
|
||||
Self::InForce => "in_force",
|
||||
Self::Faculty => "faculty",
|
||||
Self::Repo => "repo",
|
||||
Self::Cut => "cut",
|
||||
Self::Severity => "severity",
|
||||
Self::Confidence => "confidence",
|
||||
Self::Origin => "origin",
|
||||
Self::Authority => "authority",
|
||||
Self::ClaimOutcome => "claim_outcome",
|
||||
Self::Outcome => "outcome",
|
||||
}
|
||||
}
|
||||
|
||||
fn from_name(name: &str) -> Option<Self> {
|
||||
Self::ALL.into_iter().find(|alias| alias.name() == name)
|
||||
}
|
||||
|
||||
fn declared_on(self, kind: PipelineKind) -> bool {
|
||||
use PipelineKind as K;
|
||||
match self {
|
||||
Self::Root | Self::InForce | Self::Faculty => true,
|
||||
Self::Repo => {
|
||||
matches!(kind, K::Campaign | K::CutSpec | K::CutReport | K::FollowUp | K::Stewardship | K::HandOff | K::Resolution)
|
||||
}
|
||||
Self::Cut => matches!(kind, K::CutSpec | K::CutReport | K::Verdict | K::Finding | K::Resolution),
|
||||
Self::Severity | Self::Confidence | Self::Origin => kind == K::Finding,
|
||||
Self::Authority => kind == K::Ruling,
|
||||
Self::ClaimOutcome => kind == K::Verdict,
|
||||
Self::Outcome => kind == K::Resolution,
|
||||
}
|
||||
}
|
||||
|
||||
/// The closed set an enum-valued alias draws from.
|
||||
fn closed(self) -> Option<&'static [&'static str]> {
|
||||
match self {
|
||||
Self::InForce => Some(&BOOLEANS),
|
||||
Self::Faculty => Some(&FACULTIES),
|
||||
Self::Severity => Some(&SEVERITIES),
|
||||
Self::Confidence => Some(&CONFIDENCES),
|
||||
Self::Origin => Some(&ORIGINS),
|
||||
Self::Authority => Some(&AUTHORITIES),
|
||||
Self::ClaimOutcome => Some(&CLAIM_OUTCOMES),
|
||||
Self::Outcome => Some(&OUTCOMES),
|
||||
Self::Root | Self::Repo | Self::Cut => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `value` is in this alias's domain: the leaf's own grammar for
|
||||
/// the three grammar-valued aliases, exact spelling for the closed sets.
|
||||
fn admits(self, value: &str) -> bool {
|
||||
match self {
|
||||
Self::Root => Slug(value.into()).validate_slug().is_ok(),
|
||||
Self::Repo => OrgRepo(value.into()).validate_org_repo().is_ok(),
|
||||
Self::Cut => Label(value.into()).validate_label().is_ok(),
|
||||
closed => closed.closed().is_some_and(|set| set.contains(&value)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The root and the local of a key, by position in `<root>:<kind>:<local>`:
|
||||
/// the leaf derives every key that way and its own test pins the three
|
||||
/// segments. The only place the read side reads a key apart.
|
||||
pub(crate) fn root_and_local(key: &str) -> (&str, &str) {
|
||||
let mut segments = key.split(':');
|
||||
let root = segments.next().unwrap_or_default();
|
||||
let local = segments.nth(1).unwrap_or_default();
|
||||
(root, local)
|
||||
}
|
||||
|
||||
fn outcome_name(outcome: &ResolutionOutcome) -> &'static str {
|
||||
match outcome {
|
||||
ResolutionOutcome::Superseded { .. } => "Superseded",
|
||||
ResolutionOutcome::Answered { .. } => "Answered",
|
||||
ResolutionOutcome::Fixed { .. } => "Fixed",
|
||||
ResolutionOutcome::Deferred { .. } => "Deferred",
|
||||
ResolutionOutcome::Recorded { .. } => "Recorded",
|
||||
ResolutionOutcome::Withdrawn { .. } => "Withdrawn",
|
||||
}
|
||||
}
|
||||
|
||||
/// One document as the evaluator sees it: its identity, its ordinal, what it
|
||||
/// answers for each alias, its citation edges, and the view a page projects.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct SelectionRow {
|
||||
pub(crate) kind: PipelineKind,
|
||||
pub(crate) key: String,
|
||||
ordinal: u64,
|
||||
values: BTreeMap<&'static str, Vec<String>>,
|
||||
refs: Vec<(CitationRole, PipelineRef)>,
|
||||
pub(crate) view: PipelineDocumentView,
|
||||
}
|
||||
|
||||
impl SelectionRow {
|
||||
/// `base` is the first non-resolution the document reaches through its
|
||||
/// subjects: the row a resolution's `repo` and `cut` are read from.
|
||||
pub(crate) fn new(
|
||||
held: &Held,
|
||||
base: &Held,
|
||||
view: PipelineDocumentView,
|
||||
refs: Vec<(CitationRole, PipelineRef)>,
|
||||
) -> Self {
|
||||
use PipelineDocument as D;
|
||||
let mut values: BTreeMap<&'static str, Vec<String>> = BTreeMap::new();
|
||||
let mut put = |alias: Alias, found: Vec<String>| {
|
||||
values.insert(alias.name(), found);
|
||||
};
|
||||
put(Alias::Root, vec![root_and_local(&held.key).0.to_string()]);
|
||||
put(Alias::InForce, vec![matches!(view.status, crate::query::PipelineStatus::InForce).to_string()]);
|
||||
put(Alias::Faculty, vec![format!("{:?}", view.admission.provenance.faculty)]);
|
||||
put(
|
||||
Alias::Repo,
|
||||
match &base.document {
|
||||
D::Campaign(campaign) => campaign.repos.iter().map(OrgRepo::identity).collect(),
|
||||
D::CutSpec(spec) => vec![spec.repo.identity()],
|
||||
D::CutReport(report) => vec![report.repo.identity()],
|
||||
D::FollowUp(follow_up) => vec![follow_up.repo.identity()],
|
||||
D::Stewardship(stewardship) => vec![stewardship.repo.identity()],
|
||||
D::HandOff(hand_off) => vec![hand_off.repo.identity()],
|
||||
D::Target(_) | D::Question(_) | D::Ruling(_) | D::Verdict(_) | D::Finding(_) | D::Instance(_) | D::Resolution(_) => {
|
||||
Vec::new()
|
||||
}
|
||||
},
|
||||
);
|
||||
put(
|
||||
Alias::Cut,
|
||||
root_and_local(&base.key)
|
||||
.1
|
||||
.strip_prefix("cut-")
|
||||
.and_then(|rest| rest.split_once('.'))
|
||||
.map(|(label, _)| label.to_string())
|
||||
.into_iter()
|
||||
.collect(),
|
||||
);
|
||||
match &held.document {
|
||||
D::Finding(finding) => {
|
||||
put(Alias::Severity, vec![format!("{:?}", finding.severity)]);
|
||||
put(Alias::Confidence, vec![format!("{:?}", finding.confidence)]);
|
||||
put(Alias::Origin, vec![format!("{:?}", finding.origin)]);
|
||||
}
|
||||
D::Ruling(ruling) => put(Alias::Authority, vec![format!("{:?}", ruling.authority)]),
|
||||
D::Verdict(verdict) => {
|
||||
put(Alias::ClaimOutcome, verdict.claims.iter().map(|claim| format!("{:?}", claim.outcome)).collect())
|
||||
}
|
||||
D::Resolution(resolution) => put(Alias::Outcome, vec![outcome_name(&resolution.outcome).to_string()]),
|
||||
_ => {}
|
||||
}
|
||||
Self { kind: held.kind, key: held.key.clone(), ordinal: view.admission.ordinal, values, refs, view }
|
||||
}
|
||||
}
|
||||
|
||||
impl Row for SelectionRow {
|
||||
fn schema_id(&self) -> &str {
|
||||
self.kind.type_id()
|
||||
}
|
||||
|
||||
fn record_key(&self) -> &str {
|
||||
&self.key
|
||||
}
|
||||
|
||||
fn ordinal(&self) -> i64 {
|
||||
self.ordinal as i64
|
||||
}
|
||||
|
||||
fn values(&self, index: &str) -> Vec<String> {
|
||||
self.values.get(index).cloned().unwrap_or_default()
|
||||
}
|
||||
|
||||
fn number(&self, _index: &str) -> Option<String> {
|
||||
None
|
||||
}
|
||||
|
||||
fn references(&self) -> Vec<(String, String, Option<Vec<u8>>)> {
|
||||
self.refs.iter().map(|(role, target)| (role.name().to_string(), target.id.0.clone(), None)).collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// The organ's closed lists: thirteen schemas, eleven aliases, fifteen roles.
|
||||
/// A schema's name is its id, so the substrate's alias matching resolves
|
||||
/// nothing beyond exact ids.
|
||||
pub(crate) struct Vocabulary;
|
||||
|
||||
impl RowSet for Vocabulary {
|
||||
fn all_schema_ids(&self) -> Vec<String> {
|
||||
PipelineKind::ALL.iter().map(|kind| kind.type_id().to_string()).collect()
|
||||
}
|
||||
|
||||
fn schema_name(&self, schema_id: &str) -> Option<String> {
|
||||
kind_of_type(schema_id).map(|kind| kind.type_id().to_string())
|
||||
}
|
||||
|
||||
fn declared_indexes(&self, schema_id: &str) -> Vec<String> {
|
||||
let Some(kind) = kind_of_type(schema_id) else { return Vec::new() };
|
||||
Alias::ALL.into_iter().filter(|alias| alias.declared_on(kind)).map(|alias| alias.name().to_string()).collect()
|
||||
}
|
||||
|
||||
fn declared_roles(&self, schema_id: &str) -> Vec<String> {
|
||||
let Some(kind) = kind_of_type(schema_id) else { return Vec::new() };
|
||||
CitationRole::ALL.into_iter().filter(|role| role.carrier() == kind).map(|role| role.name().to_string()).collect()
|
||||
}
|
||||
|
||||
fn is_numeric(&self, _schema_id: &str, _index: &str) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
fn target_leaves(&self, role: &str) -> Vec<String> {
|
||||
CitationRole::from_name(role)
|
||||
.map(|role| role.target_kinds().into_iter().map(|kind| kind.type_id().to_string()).collect())
|
||||
.unwrap_or_default()
|
||||
}
|
||||
}
|
||||
|
||||
fn invalid(field: impl Into<String>, value: &str, message: impl Into<String>) -> MindRefusal {
|
||||
MindRefusal::SelectionInvalid { field: field.into(), value: Some(value.to_string()), message: message.into() }
|
||||
}
|
||||
|
||||
/// The organ's door (S6), run after `cultnet_rs::validate` has accepted the
|
||||
/// selection's names: it refuses a value the vocabulary's domains do not hold,
|
||||
/// so a malformed filter never reads as an empty answer. It also returns the
|
||||
/// selection with every `repo` value in the identity spelling the rows carry,
|
||||
/// because `OrgRepo` identity is case-insensitive.
|
||||
///
|
||||
/// Refuses, naming the field the client sent: a `schemas` entry that is not
|
||||
/// one of the thirteen ids; a `keys` entry that is not a full id whose kind
|
||||
/// segment reads and passes the leaf's ref grammar; an `any_of` value outside
|
||||
/// its alias's domain; and a `cites.target` whose key fails the ref grammar
|
||||
/// for its schema's kind (V24's rule, at this door). Whether the target
|
||||
/// schema id is known is the substrate's refusal and is not repeated here.
|
||||
pub(crate) fn refuse_values(selection: &Selection) -> Result<Selection, MindRefusal> {
|
||||
for schema in selection.schemas.iter().flatten() {
|
||||
if kind_of_type(schema).is_none() {
|
||||
return Err(invalid("schemas", schema, "no such schema in this mind"));
|
||||
}
|
||||
}
|
||||
for key in selection.keys.iter().flatten() {
|
||||
let Some(kind) = kind_of_id(key) else {
|
||||
return Err(invalid("keys", key, "not a pipeline id: the kind segment does not read"));
|
||||
};
|
||||
if let Err(refusal) = (PipelineRef { kind, id: key.as_str().into() }).validate_ref() {
|
||||
return Err(invalid("keys", key, format!("not a pipeline id: {refusal:?}")));
|
||||
}
|
||||
}
|
||||
let mut canonical = selection.clone();
|
||||
for (position, predicate) in canonical.fields.iter_mut().flatten().enumerate() {
|
||||
let Some(alias) = Alias::from_name(&predicate.index) else { continue };
|
||||
for value in predicate.values.iter_mut().flatten() {
|
||||
if !alias.admits(value) {
|
||||
return Err(invalid(
|
||||
format!("fields[{position}].values"),
|
||||
value,
|
||||
match alias.closed() {
|
||||
Some(set) => format!("{} takes one of {}", alias.name(), set.join(", ")),
|
||||
None => format!("not a valid {}", alias.name()),
|
||||
},
|
||||
));
|
||||
}
|
||||
if alias == Alias::Repo {
|
||||
*value = OrgRepo(value.clone()).identity();
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(cites) = &selection.cites
|
||||
&& let Some(kind) = kind_of_type(&cites.target.schema_id)
|
||||
&& let Err(refusal) = (PipelineRef { kind, id: cites.target.record_key.as_str().into() }).validate_ref()
|
||||
{
|
||||
return Err(invalid("cites.target", &cites.target.record_key, format!("not a {} id: {refusal:?}", kind.name())));
|
||||
}
|
||||
Ok(canonical)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use PipelineKind as K;
|
||||
|
||||
fn sorted(mut names: Vec<String>) -> Vec<String> {
|
||||
names.sort();
|
||||
names
|
||||
}
|
||||
|
||||
fn strings(names: &[&str]) -> Vec<String> {
|
||||
sorted(names.iter().map(|name| name.to_string()).collect())
|
||||
}
|
||||
|
||||
/// The vocabulary's tables, read back through the substrate's trait: every
|
||||
/// kind declares the three universal aliases and exactly the ones its
|
||||
/// documents carry, and no alias is numeric, so a comparison is always
|
||||
/// refused at the substrate's door.
|
||||
#[test]
|
||||
fn each_kind_declares_exactly_its_aliases_and_none_is_numeric() {
|
||||
let table: [(K, &[&str]); 13] = [
|
||||
(K::Campaign, &["repo"]),
|
||||
(K::Target, &[]),
|
||||
(K::Question, &[]),
|
||||
(K::Ruling, &["authority"]),
|
||||
(K::CutSpec, &["repo", "cut"]),
|
||||
(K::CutReport, &["repo", "cut"]),
|
||||
(K::Verdict, &["cut", "claim_outcome"]),
|
||||
(K::Finding, &["cut", "severity", "confidence", "origin"]),
|
||||
(K::FollowUp, &["repo"]),
|
||||
(K::Resolution, &["repo", "cut", "outcome"]),
|
||||
(K::Instance, &[]),
|
||||
(K::Stewardship, &["repo"]),
|
||||
(K::HandOff, &["repo"]),
|
||||
];
|
||||
for (kind, extra) in table {
|
||||
let want = [&["root", "in_force", "faculty"][..], extra].concat();
|
||||
assert_eq!(sorted(Vocabulary.declared_indexes(kind.type_id())), strings(&want), "{kind:?}");
|
||||
for alias in Alias::ALL {
|
||||
assert!(!Vocabulary.is_numeric(kind.type_id(), alias.name()), "{kind:?}/{}", alias.name());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Each role is declared by the one kind whose field carries it, and its
|
||||
/// targets are the kinds that field is typed to, "any" spelled as all
|
||||
/// thirteen.
|
||||
#[test]
|
||||
fn each_kind_declares_the_roles_it_carries_and_their_targets() {
|
||||
let table: [(K, &[&str]); 13] = [
|
||||
(K::Campaign, &[]),
|
||||
(K::Target, &[]),
|
||||
(K::Question, &["raised_in"]),
|
||||
(K::Ruling, &["answers"]),
|
||||
(K::CutSpec, &["rulings", "questions"]),
|
||||
(K::CutReport, &["cut_spec", "forks"]),
|
||||
(K::Verdict, &["cut_report", "findings"]),
|
||||
(K::Finding, &["verdict"]),
|
||||
(K::FollowUp, &["source"]),
|
||||
(K::Resolution, &["subject", "superseded_by", "resolved_by", "deferred_to"]),
|
||||
(K::Instance, &[]),
|
||||
(K::Stewardship, &[]),
|
||||
(K::HandOff, &["documents"]),
|
||||
];
|
||||
for (kind, roles) in table {
|
||||
assert_eq!(sorted(Vocabulary.declared_roles(kind.type_id())), strings(roles), "{kind:?}");
|
||||
}
|
||||
assert_eq!(Vocabulary.target_leaves("answers"), vec![K::Question.type_id().to_string()]);
|
||||
assert_eq!(Vocabulary.target_leaves("rulings"), vec![K::Ruling.type_id().to_string()]);
|
||||
assert_eq!(Vocabulary.target_leaves("subject").len(), 13);
|
||||
assert!(Vocabulary.target_leaves("nonsense").is_empty());
|
||||
assert_eq!(Vocabulary.schema_name(K::Question.type_id()), Some(K::Question.type_id().to_string()));
|
||||
assert_eq!(Vocabulary.schema_name("epiphany.pipeline.nonsense.v2"), None);
|
||||
}
|
||||
}
|
||||
228
crates/huginn-mind/src/store.rs
Normal file
228
crates/huginn-mind/src/store.rs
Normal file
|
|
@ -0,0 +1,228 @@
|
|||
//! The store port. `cultcache-rs` exposes compare-and-swap as inherent methods
|
||||
//! on its concrete stores, not on `CacheBackingStore`, so the one narrow trait
|
||||
//! here is how the commit primitive gets it injected: the owned redb store in
|
||||
//! the daemon, a memory store in every admission test, a refusing store in the
|
||||
//! atomicity test. Nothing else is abstracted.
|
||||
|
||||
use anyhow::Result;
|
||||
use cultcache_rs::CultCacheEnvelope;
|
||||
|
||||
/// The one store a daemon opens. Re-exported so the crate that serves a mind
|
||||
/// names the store type without declaring `cultcache-rs` a second time: one
|
||||
/// dependency entry, one rev, one place to move it.
|
||||
pub use cultcache_rs::OwnedRedbMessagePackBackingStore;
|
||||
|
||||
/// CultCache's own row traits, re-exported for the same reason: the crate
|
||||
/// that serves a mind reaches a store's rows through here rather than
|
||||
/// declaring `cultcache-rs` a second time. The daemon's store-integrity tests
|
||||
/// use it to break a row on purpose, over a real store, so a read over the
|
||||
/// broken store raises the refusal its dispatch arm must carry — but that is
|
||||
/// not the only door a row already has: `MindStore`'s own supertrait is
|
||||
/// `CacheBackingStore`, and its public `compare_and_swap_batch` already lets
|
||||
/// non-test code outside this crate write a row directly. Sealing the store
|
||||
/// so admission is its only write path is a follow-up, its own cut; this
|
||||
/// re-export does not widen anything that door does not already reach. It
|
||||
/// carries two traits, `CacheBackingStore` and `DatabaseEntry`, plus
|
||||
/// `DatabaseEntry`'s own derive macro, which shares its name in a different
|
||||
/// namespace and rides along unnamed.
|
||||
pub use cultcache_rs::{CacheBackingStore, DatabaseEntry};
|
||||
|
||||
/// A store a mind can be opened over. `Clone` shares ownership of the same
|
||||
/// store (the owned redb store's clones share its lock and handle), which is
|
||||
/// how the cache image and the commit path hold one store between them.
|
||||
pub trait MindStore: CacheBackingStore + Clone + 'static {
|
||||
/// One transactional step: every `expected` row must be stored exactly as
|
||||
/// given, no `replacements` row outside `expected` may already exist, and
|
||||
/// then every replacement lands, or nothing does and `Ok(false)` says so.
|
||||
/// The store's own rule that every expected identity is also replaced is
|
||||
/// kept: a strong read is re-inserted unchanged beside the writes.
|
||||
fn compare_and_swap_batch(
|
||||
&self,
|
||||
expected: &[CultCacheEnvelope],
|
||||
replacements: Vec<CultCacheEnvelope>,
|
||||
) -> Result<bool>;
|
||||
}
|
||||
|
||||
impl MindStore for OwnedRedbMessagePackBackingStore {
|
||||
fn compare_and_swap_batch(
|
||||
&self,
|
||||
expected: &[CultCacheEnvelope],
|
||||
replacements: Vec<CultCacheEnvelope>,
|
||||
) -> Result<bool> {
|
||||
OwnedRedbMessagePackBackingStore::compare_and_swap_batch(self, expected, replacements)
|
||||
}
|
||||
}
|
||||
|
||||
/// Test stores: a memory store with the redb store's batch semantics, and one
|
||||
/// that can be told to lose or fail its next swap.
|
||||
#[cfg(test)]
|
||||
pub(crate) mod test_stores {
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
use cultcache_rs::{CacheBackingStore, CultCacheEnvelope, PushAllOptions};
|
||||
|
||||
use super::MindStore;
|
||||
|
||||
type Identity = (String, String);
|
||||
|
||||
fn identity(entry: &CultCacheEnvelope) -> Identity {
|
||||
(entry.r#type.clone(), entry.key.clone())
|
||||
}
|
||||
|
||||
fn unique(entries: &[CultCacheEnvelope], label: &str) -> Result<BTreeSet<Identity>> {
|
||||
let ids = entries.iter().map(identity).collect::<BTreeSet<_>>();
|
||||
if ids.len() != entries.len() {
|
||||
return Err(anyhow!("conditional batch {label} set contains duplicate identities"));
|
||||
}
|
||||
Ok(ids)
|
||||
}
|
||||
|
||||
/// One row per `(type, key)` behind a mutex; clones share it. Counts
|
||||
/// `pull_all` calls so the opener test can prove nothing attached.
|
||||
#[derive(Clone, Default)]
|
||||
pub(crate) struct MemoryStore {
|
||||
rows: Arc<Mutex<BTreeMap<Identity, CultCacheEnvelope>>>,
|
||||
pulls: Arc<AtomicUsize>,
|
||||
}
|
||||
|
||||
impl MemoryStore {
|
||||
pub(crate) fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
pub(crate) fn pull_count(&self) -> usize {
|
||||
self.pulls.load(Ordering::SeqCst)
|
||||
}
|
||||
|
||||
/// Every row, in identity order: the bytes a test compares before
|
||||
/// and after a refused admission.
|
||||
pub(crate) fn rows(&self) -> Vec<CultCacheEnvelope> {
|
||||
self.rows.lock().unwrap().values().cloned().collect()
|
||||
}
|
||||
|
||||
/// Writes a row without the commit path, for the opener tests and the
|
||||
/// one replay-planting test.
|
||||
pub(crate) fn plant(&self, entry: CultCacheEnvelope) {
|
||||
self.rows.lock().unwrap().insert(identity(&entry), entry);
|
||||
}
|
||||
}
|
||||
|
||||
impl CacheBackingStore for MemoryStore {
|
||||
fn pull_all(&self) -> Result<Vec<CultCacheEnvelope>> {
|
||||
self.pulls.fetch_add(1, Ordering::SeqCst);
|
||||
Ok(self.rows())
|
||||
}
|
||||
|
||||
fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()> {
|
||||
self.plant(entry.clone());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn delete(&mut self, entry: &CultCacheEnvelope) -> Result<()> {
|
||||
self.rows.lock().unwrap().remove(&identity(entry));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn push_all(&mut self, entries: &[CultCacheEnvelope], _options: PushAllOptions) -> Result<()> {
|
||||
*self.rows.lock().unwrap() = entries.iter().map(|entry| (identity(entry), entry.clone())).collect();
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl MindStore for MemoryStore {
|
||||
fn compare_and_swap_batch(
|
||||
&self,
|
||||
expected: &[CultCacheEnvelope],
|
||||
replacements: Vec<CultCacheEnvelope>,
|
||||
) -> Result<bool> {
|
||||
if replacements.is_empty() {
|
||||
return Err(anyhow!("conditional batch requires a non-empty replacement set"));
|
||||
}
|
||||
let expected_ids = unique(expected, "expected")?;
|
||||
let replacement_ids = unique(&replacements, "replacement")?;
|
||||
if !expected_ids.is_subset(&replacement_ids) {
|
||||
return Err(anyhow!("conditional batch must replace every expected identity"));
|
||||
}
|
||||
let mut rows = self.rows.lock().unwrap();
|
||||
for row in expected {
|
||||
if rows.get(&identity(row)) != Some(row) {
|
||||
return Ok(false);
|
||||
}
|
||||
}
|
||||
for row in &replacements {
|
||||
if !expected_ids.contains(&identity(row)) && rows.contains_key(&identity(row)) {
|
||||
return Ok(false);
|
||||
}
|
||||
}
|
||||
for row in replacements {
|
||||
rows.insert(identity(&row), row);
|
||||
}
|
||||
Ok(true)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum SwapCommand {
|
||||
Delegate,
|
||||
Lose,
|
||||
Fail,
|
||||
}
|
||||
|
||||
/// A memory store whose next swap can be commanded to lose (`Ok(false)`)
|
||||
/// or fail (`Err`), so the commit path's two non-committing exits are
|
||||
/// reachable without a second process.
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct RefusingStore {
|
||||
inner: MemoryStore,
|
||||
command: Arc<Mutex<SwapCommand>>,
|
||||
}
|
||||
|
||||
impl RefusingStore {
|
||||
pub(crate) fn new() -> Self {
|
||||
Self { inner: MemoryStore::new(), command: Arc::new(Mutex::new(SwapCommand::Delegate)) }
|
||||
}
|
||||
|
||||
pub(crate) fn command(&self, command: SwapCommand) {
|
||||
*self.command.lock().unwrap() = command;
|
||||
}
|
||||
|
||||
pub(crate) fn rows(&self) -> Vec<CultCacheEnvelope> {
|
||||
self.inner.rows()
|
||||
}
|
||||
}
|
||||
|
||||
impl CacheBackingStore for RefusingStore {
|
||||
fn pull_all(&self) -> Result<Vec<CultCacheEnvelope>> {
|
||||
self.inner.pull_all()
|
||||
}
|
||||
|
||||
fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()> {
|
||||
self.inner.push(entry)
|
||||
}
|
||||
|
||||
fn delete(&mut self, entry: &CultCacheEnvelope) -> Result<()> {
|
||||
self.inner.delete(entry)
|
||||
}
|
||||
|
||||
fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> Result<()> {
|
||||
self.inner.push_all(entries, options)
|
||||
}
|
||||
}
|
||||
|
||||
impl MindStore for RefusingStore {
|
||||
fn compare_and_swap_batch(
|
||||
&self,
|
||||
expected: &[CultCacheEnvelope],
|
||||
replacements: Vec<CultCacheEnvelope>,
|
||||
) -> Result<bool> {
|
||||
match *self.command.lock().unwrap() {
|
||||
SwapCommand::Delegate => self.inner.compare_and_swap_batch(expected, replacements),
|
||||
SwapCommand::Lose => Ok(false),
|
||||
SwapCommand::Fail => Err(anyhow!("the store refused the swap on command")),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
287
crates/huginn-mind/src/wire.rs
Normal file
287
crates/huginn-mind/src/wire.rs
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
//! The organ's request and response vocabulary: one operation per `Mind`
|
||||
//! method and `whoami`. Payloads are the admission and read types unchanged.
|
||||
//! The daemon carries these in CultNet's operation envelope; the client
|
||||
//! constructs them. Nothing here validates a document, derives a status or
|
||||
//! names a transport.
|
||||
|
||||
use cultcache_rs::DatabaseEntry;
|
||||
use cultnet_rs::{CultMeshCdnArtifactManifest, Selection};
|
||||
use epiphany_pipeline::{PIPELINE_SCHEMA_EPOCH, PipelineKind, PipelineRef, Slug};
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::admission::{PipelineAdmissionBatch, PipelineAdmissionOutcome};
|
||||
use crate::mind::Mind;
|
||||
use crate::query::{PipelineDocumentView, PipelineSelectionPage, SemanticQuery};
|
||||
use crate::receipt::HuginnCommitReceipt;
|
||||
use crate::refusal::MindRefusal;
|
||||
use crate::store::MindStore;
|
||||
|
||||
/// The envelope's `service_id`: one service, one mind, whichever mind the
|
||||
/// process opened.
|
||||
pub const MIND_SERVICE_ID: &str = "huginn.mind";
|
||||
pub const MIND_REQUEST_SCHEMA: &str = "huginn.mind_request.v1";
|
||||
pub const MIND_RESPONSE_SCHEMA: &str = "huginn.mind_response.v1";
|
||||
|
||||
/// The largest answer that is deferred rather than refused, measured on the
|
||||
/// encoded payload. The daemon refuses past it and a client caps its fetch at
|
||||
/// it, so both sides read one number.
|
||||
pub const MAX_DEFERRED_BODY_BYTES: u64 = 64 * 1024 * 1024;
|
||||
|
||||
/// The published JSON schema of each wire type, as the catalog advertises it.
|
||||
/// The files are derived from the types below and pinned byte for byte by
|
||||
/// `published_wire_schemas_match_derivation`, so the catalog cannot advertise
|
||||
/// a shape the crate does not speak.
|
||||
pub const MIND_REQUEST_SCHEMA_JSON: &str =
|
||||
include_str!("../../../schemas/cultnet/huginn.mind_request.v1.schema.json");
|
||||
pub const MIND_RESPONSE_SCHEMA_JSON: &str =
|
||||
include_str!("../../../schemas/cultnet/huginn.mind_response.v1.schema.json");
|
||||
|
||||
/// The request's `selection` is CultNet's, and its published schema is
|
||||
/// CultLib's: a `$ref` to the `$id` it publishes, so one owner holds the
|
||||
/// selection's shape and nothing here copies it.
|
||||
fn selection_ref(_: &mut schemars::SchemaGenerator) -> schemars::Schema {
|
||||
schemars::json_schema!({
|
||||
"$ref": "https://github.com/GameCult/CultLib/contracts/cultnet/cultnet.selection.schema.json"
|
||||
})
|
||||
}
|
||||
|
||||
/// The deferred answer's manifest is CultMesh's own type, a positional array
|
||||
/// the C# reference defines (`CultMeshCdnArtifactManifest`, schema version
|
||||
/// `gamecult.mesh.cdn_artifact_manifest.v1`). CultLib publishes no JSON schema
|
||||
/// for it, so the schema says what the reference says: an array of its ten
|
||||
/// fields in order, and nothing here copies their meaning.
|
||||
fn manifest_array(_: &mut schemars::SchemaGenerator) -> schemars::Schema {
|
||||
schemars::json_schema!({
|
||||
"description": "gamecult.mesh.cdn_artifact_manifest.v1, the C# reference's CultMeshCdnArtifactManifest as its positional array: [artifactId, kind, version, contentHash, sizeBytes, mimeType, createdAtUtc, chunks[[chunkHash, offset, sizeBytes, recordKey]], tags, metadata].",
|
||||
"type": "array"
|
||||
})
|
||||
}
|
||||
|
||||
/// Every operation a mind answers. `Admit` carries the batch whole because the
|
||||
/// batch already names the instance and the asker; a second `instance` beside
|
||||
/// it would be two declarations. Every read names the instance because ruling
|
||||
/// 14 refuses another mind's identity whatever the transport, and a read of
|
||||
/// the wrong mind is that same collision on the read side. `Whoami` names
|
||||
/// none: it is how a client learns which mind it reached.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub enum HuginnMindRequest {
|
||||
Whoami,
|
||||
Admit(PipelineAdmissionBatch),
|
||||
View { instance: Slug, id: PipelineRef },
|
||||
Query {
|
||||
instance: Slug,
|
||||
#[schemars(schema_with = "selection_ref")]
|
||||
selection: Selection,
|
||||
semantic: Option<SemanticQuery>,
|
||||
},
|
||||
}
|
||||
|
||||
impl HuginnMindRequest {
|
||||
/// The envelope's `operation`.
|
||||
pub fn operation(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Whoami => "whoami",
|
||||
Self::Admit(_) => "admit",
|
||||
Self::View { .. } => "view",
|
||||
Self::Query { .. } => "query",
|
||||
}
|
||||
}
|
||||
|
||||
/// The instance the request declares; `Whoami` declares none.
|
||||
pub fn instance(&self) -> Option<&Slug> {
|
||||
match self {
|
||||
Self::Whoami => None,
|
||||
Self::Admit(batch) => Some(&batch.instance),
|
||||
Self::View { instance, .. } | Self::Query { instance, .. } => Some(instance),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Every answer a mind gives. A refusal is data in the response, never a
|
||||
/// transport failure: the read side carries `Refused`, the write side carries
|
||||
/// the outcome's own `Refused`.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub enum HuginnMindResponse {
|
||||
Whoami(MindStatus),
|
||||
Admit(PipelineAdmissionOutcome),
|
||||
View(Option<PipelineDocumentView>),
|
||||
Query(PipelineSelectionPage),
|
||||
Refused(MindRefusal),
|
||||
/// The answer did not fit one send and is being held for the client to
|
||||
/// fetch. Nothing in a mind produces this: the daemon's serve module does,
|
||||
/// in place of an answer, and the answer it stands for is the named
|
||||
/// MessagePack of the `HuginnMindResponse` the mind gave.
|
||||
Deferred(DeferredAnswer),
|
||||
}
|
||||
|
||||
/// Where the body of an answer that did not fit one send can be fetched: the
|
||||
/// reference's own manifest, whose `contentHash` is the SHA-256 of the answer's
|
||||
/// encoded bytes and whose chunks are asked for, one at a time, on the session
|
||||
/// this arrived on. Row-level facts come from asking again with a header
|
||||
/// projection; there is no summary.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct DeferredAnswer {
|
||||
#[schemars(schema_with = "manifest_array")]
|
||||
pub manifest: CultMeshCdnArtifactManifest,
|
||||
}
|
||||
|
||||
impl HuginnMindResponse {
|
||||
/// The envelope's `status`, in the C# reference's vocabulary: `rejected`
|
||||
/// exactly for a refusal, `accepted` otherwise. `Conflict` and
|
||||
/// `AlreadyAdmitted` are answers about the mind, not refusals of the
|
||||
/// request. A `Deferred` is nominally `accepted`; the daemon sets the
|
||||
/// envelope of a deferred answer to the status of the answer it stands for.
|
||||
pub fn status(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Refused(_) | Self::Admit(PipelineAdmissionOutcome::Refused(_)) => "rejected",
|
||||
_ => "accepted",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// How far the semantic index has caught up with the mind. The mind decides
|
||||
/// none of it: the daemon's index worker reports it, in memory, and a restart
|
||||
/// recomputes it. `pending` counts documents not yet known to be in the index.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub enum IndexStatus {
|
||||
/// The worker has not yet compared the mind with the collection.
|
||||
Reconciling { pending: u32 },
|
||||
/// Every indexable document is in the collection.
|
||||
Current,
|
||||
/// Reconciled, and indexing what is still pending.
|
||||
Behind { pending: u32 },
|
||||
/// The last attempt failed (the embedder or the vector store is
|
||||
/// unreachable, or refused); it is retried with a backoff and nothing is
|
||||
/// dropped. `attempts` counts consecutive failures.
|
||||
Failing { pending: u32, attempts: u32, error: String },
|
||||
/// The collection belongs to another writer. The worker never recreates
|
||||
/// it; it re-reads it on the same backoff.
|
||||
Refused { pending: u32, attempts: u32, reason: String },
|
||||
}
|
||||
|
||||
/// What a mind says about itself: the typed state a dashboard projects.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct MindStatus {
|
||||
pub instance: Slug,
|
||||
pub schema_epoch: String,
|
||||
/// Pipeline documents in the image; the epoch record and the receipts are
|
||||
/// not documents and are not counted here.
|
||||
pub documents: u32,
|
||||
pub receipts: u32,
|
||||
pub index: IndexStatus,
|
||||
}
|
||||
|
||||
impl<S: MindStore> Mind<S> {
|
||||
/// `whoami`, derived from the image every time it is asked, and from the
|
||||
/// index status the caller holds: the mind stores neither.
|
||||
pub fn status(&self, index: IndexStatus) -> MindStatus {
|
||||
let mut documents = 0_u32;
|
||||
let mut receipts = 0_u32;
|
||||
for envelope in self.envelopes() {
|
||||
if envelope.r#type == HuginnCommitReceipt::TYPE {
|
||||
receipts += 1;
|
||||
} else if PipelineKind::ALL.iter().any(|kind| kind.type_id() == envelope.r#type) {
|
||||
documents += 1;
|
||||
}
|
||||
}
|
||||
MindStatus {
|
||||
instance: self.instance().clone(),
|
||||
schema_epoch: PIPELINE_SCHEMA_EPOCH.to_string(),
|
||||
documents,
|
||||
receipts,
|
||||
index,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::query::PipelinePageItems;
|
||||
use crate::fixtures::{CAMPAIGN, INSTANCE, OTHER_INSTANCE, instance, provenance, r, seeded, slug};
|
||||
use crate::receipt::Faculty;
|
||||
|
||||
fn requests() -> Vec<HuginnMindRequest> {
|
||||
let batch = PipelineAdmissionBatch {
|
||||
instance: slug(INSTANCE),
|
||||
provenance: provenance(Faculty::Hands),
|
||||
documents: vec![instance(INSTANCE)],
|
||||
};
|
||||
vec![
|
||||
HuginnMindRequest::Whoami,
|
||||
HuginnMindRequest::Admit(batch),
|
||||
HuginnMindRequest::View { instance: slug(INSTANCE), id: r(PipelineKind::Campaign, CAMPAIGN) },
|
||||
HuginnMindRequest::Query { instance: slug(OTHER_INSTANCE), selection: Selection::default(), semantic: None },
|
||||
]
|
||||
}
|
||||
|
||||
fn responses() -> Vec<HuginnMindResponse> {
|
||||
let refusal = MindRefusal::ForeignInstance { declared: OTHER_INSTANCE.into(), mind: INSTANCE.into() };
|
||||
vec![
|
||||
HuginnMindResponse::Whoami(seeded().status(IndexStatus::Current)),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::AlreadyAdmitted { receipt_id: "r1".into() }),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Conflict { identities: vec![] }),
|
||||
HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(refusal.clone())),
|
||||
HuginnMindResponse::View(None),
|
||||
HuginnMindResponse::Query(PipelineSelectionPage {
|
||||
matched: 0,
|
||||
as_of: 1,
|
||||
next: None,
|
||||
items: PipelinePageItems::Headers(vec![]),
|
||||
edges: None,
|
||||
}),
|
||||
HuginnMindResponse::Refused(refusal),
|
||||
HuginnMindResponse::Deferred(DeferredAnswer {
|
||||
manifest: cultnet_rs::pack_content("huginn.mind_response", "package", "", "", "", b"body", 4)
|
||||
.unwrap()
|
||||
.0,
|
||||
}),
|
||||
]
|
||||
}
|
||||
|
||||
/// The vocabulary is the mind's methods, the declared instance is on every
|
||||
/// request that names a mind, and the envelope's status is derived from
|
||||
/// the response rather than chosen by whoever sends it.
|
||||
#[test]
|
||||
fn the_wire_vocabulary_is_the_minds_methods_and_status_is_derived_from_the_response() {
|
||||
let names: Vec<&str> = requests().iter().map(|request| request.operation()).collect();
|
||||
assert_eq!(names, ["whoami", "admit", "view", "query"]);
|
||||
for request in requests() {
|
||||
let declared = request.instance().is_none();
|
||||
assert_eq!(declared, matches!(request, HuginnMindRequest::Whoami), "{request:?}");
|
||||
let bytes = rmp_serde::to_vec_named(&request).unwrap();
|
||||
assert_eq!(rmp_serde::from_slice::<HuginnMindRequest>(&bytes).unwrap(), request);
|
||||
}
|
||||
assert_eq!(
|
||||
requests()[3].instance(),
|
||||
Some(&slug(OTHER_INSTANCE)),
|
||||
"the request's instance is the one it declares, not the mind's"
|
||||
);
|
||||
|
||||
for response in responses() {
|
||||
let rejected = matches!(
|
||||
response,
|
||||
HuginnMindResponse::Refused(_) | HuginnMindResponse::Admit(PipelineAdmissionOutcome::Refused(_))
|
||||
);
|
||||
assert_eq!(response.status(), if rejected { "rejected" } else { "accepted" }, "{response:?}");
|
||||
let bytes = rmp_serde::to_vec_named(&response).unwrap();
|
||||
assert_eq!(rmp_serde::from_slice::<HuginnMindResponse>(&bytes).unwrap(), response);
|
||||
}
|
||||
}
|
||||
|
||||
/// The two files the catalog publishes are the derivation of the two
|
||||
/// types, byte for byte.
|
||||
#[test]
|
||||
fn published_wire_schemas_match_derivation() {
|
||||
let request = serde_json::to_string_pretty(&schemars::schema_for!(HuginnMindRequest)).unwrap();
|
||||
let response = serde_json::to_string_pretty(&schemars::schema_for!(HuginnMindResponse)).unwrap();
|
||||
assert_eq!(MIND_REQUEST_SCHEMA_JSON.replace("\r\n", "\n"), request);
|
||||
assert_eq!(MIND_RESPONSE_SCHEMA_JSON.replace("\r\n", "\n"), response);
|
||||
// The selection is CultLib's, referenced by the `$id` it publishes and
|
||||
// not copied into this schema.
|
||||
let selection = "https://github.com/GameCult/CultLib/contracts/cultnet/cultnet.selection.schema.json";
|
||||
assert!(request.contains(&format!("\"$ref\": \"{selection}\"")));
|
||||
assert!(!request.contains("\"cursor\""), "no second copy of the selection's shape");
|
||||
}
|
||||
}
|
||||
3
dist/cli.d.ts
vendored
3
dist/cli.d.ts
vendored
|
|
@ -1,3 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
export {};
|
||||
//# sourceMappingURL=cli.d.ts.map
|
||||
1
dist/cli.d.ts.map
vendored
1
dist/cli.d.ts.map
vendored
|
|
@ -1 +0,0 @@
|
|||
{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":""}
|
||||
21
dist/cli.js
vendored
21
dist/cli.js
vendored
|
|
@ -1,21 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
"use strict";
|
||||
Object.defineProperty(exports, "__esModule", { value: true });
|
||||
const promises_1 = require("node:fs/promises");
|
||||
const inspection_1 = require("cultcache-ts/inspection");
|
||||
const huginn_eve_dsl_1 = require("./huginn-eve-dsl");
|
||||
async function main(args) {
|
||||
const filePath = args[0];
|
||||
if (!filePath || args.includes("--help") || args.includes("-h")) {
|
||||
process.stderr.write("Usage: huginn <state.cc|state.msgpack|state.mpack>\n");
|
||||
process.exitCode = filePath ? 0 : 1;
|
||||
return;
|
||||
}
|
||||
const bytes = await (0, promises_1.readFile)(filePath);
|
||||
const inspection = (0, inspection_1.inspectCultCacheBytes)(filePath, bytes);
|
||||
process.stdout.write(`${(0, huginn_eve_dsl_1.buildHuginnEveDsl)(inspection)}\n`);
|
||||
}
|
||||
void main(process.argv.slice(2)).catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
3
dist/huginn-eve-dsl.d.ts
vendored
3
dist/huginn-eve-dsl.d.ts
vendored
|
|
@ -1,3 +0,0 @@
|
|||
import type { CultCacheInspection } from "cultcache-ts/inspection";
|
||||
export declare function buildHuginnEveDsl(inspection: CultCacheInspection): string;
|
||||
//# sourceMappingURL=huginn-eve-dsl.d.ts.map
|
||||
1
dist/huginn-eve-dsl.d.ts.map
vendored
1
dist/huginn-eve-dsl.d.ts.map
vendored
|
|
@ -1 +0,0 @@
|
|||
{"version":3,"file":"huginn-eve-dsl.d.ts","sourceRoot":"","sources":["../src/huginn-eve-dsl.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,mBAAmB,EAA0C,MAAM,yBAAyB,CAAC;AAE3G,wBAAgB,iBAAiB,CAAC,UAAU,EAAE,mBAAmB,GAAG,MAAM,CA8BzE"}
|
||||
55
dist/huginn-eve-dsl.js
vendored
55
dist/huginn-eve-dsl.js
vendored
|
|
@ -1,55 +0,0 @@
|
|||
"use strict";
|
||||
Object.defineProperty(exports, "__esModule", { value: true });
|
||||
exports.buildHuginnEveDsl = buildHuginnEveDsl;
|
||||
function buildHuginnEveDsl(inspection) {
|
||||
return [
|
||||
"# Huginn emits Eve DSL. Renderers compile and lower it.",
|
||||
`surface cultcache.huginn.inspector ${quoteDsl("Huginn .cc Inspector")}`,
|
||||
"version 1",
|
||||
"",
|
||||
"collection huginn.catalog.entries",
|
||||
...inspection.catalog.map((entry) => `item huginn.catalog.entries ${quoteDsl(renderCatalogEntry(entry))}`),
|
||||
"",
|
||||
"collection huginn.records.entries",
|
||||
...inspection.records.map((record) => `item huginn.records.entries ${quoteDsl(renderRecord(record))}`),
|
||||
"",
|
||||
"card source \"Source Witness\"",
|
||||
`text ${quoteDsl(`file: ${inspection.filePath}`)}`,
|
||||
`text ${quoteDsl(`format: ${inspection.format}`)}`,
|
||||
`text ${quoteDsl(`bytes: ${inspection.fileSizeBytes}`)}`,
|
||||
"text \".cc bytes are authority; Huginn only emits Eve DSL.\"",
|
||||
"end",
|
||||
"",
|
||||
"card catalog \"Schema Catalog\"",
|
||||
`text ${quoteDsl(`schemas: ${inspection.catalog.length}`)}`,
|
||||
"list \"Catalog Entries\" bind huginn.catalog.entries",
|
||||
"end",
|
||||
"",
|
||||
"card records \"Persisted Records\"",
|
||||
`text ${quoteDsl(`records: ${inspection.records.length}`)}`,
|
||||
"list \"Record Entries\" bind huginn.records.entries",
|
||||
"end",
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
function renderCatalogEntry(entry) {
|
||||
return [
|
||||
entry.schemaName,
|
||||
entry.schemaVersion,
|
||||
entry.schemaId,
|
||||
`${entry.members.length} members`,
|
||||
].join(" | ");
|
||||
}
|
||||
function renderRecord(record) {
|
||||
const status = record.payloadDecodeError ? `decode error: ${record.payloadDecodeError}` : "decoded";
|
||||
return [
|
||||
record.key,
|
||||
record.schemaName,
|
||||
record.storedAt,
|
||||
`${record.payloadBytes} bytes`,
|
||||
status,
|
||||
].join(" | ");
|
||||
}
|
||||
function quoteDsl(value) {
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
4
dist/index.d.ts
vendored
4
dist/index.d.ts
vendored
|
|
@ -1,4 +0,0 @@
|
|||
export { buildHuginnEveDsl } from "./huginn-eve-dsl";
|
||||
export { inspectCultCacheBytes } from "cultcache-ts/inspection";
|
||||
export type { CultCacheInspection, InspectedCatalogEntry, InspectedCatalogMember, InspectedRecord, } from "cultcache-ts/inspection";
|
||||
//# sourceMappingURL=index.d.ts.map
|
||||
1
dist/index.d.ts.map
vendored
1
dist/index.d.ts.map
vendored
|
|
@ -1 +0,0 @@
|
|||
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,kBAAkB,CAAC;AACrD,OAAO,EAAE,qBAAqB,EAAE,MAAM,yBAAyB,CAAC;AAChE,YAAY,EACV,mBAAmB,EACnB,qBAAqB,EACrB,sBAAsB,EACtB,eAAe,GAChB,MAAM,yBAAyB,CAAC"}
|
||||
7
dist/index.js
vendored
7
dist/index.js
vendored
|
|
@ -1,7 +0,0 @@
|
|||
"use strict";
|
||||
Object.defineProperty(exports, "__esModule", { value: true });
|
||||
exports.inspectCultCacheBytes = exports.buildHuginnEveDsl = void 0;
|
||||
var huginn_eve_dsl_1 = require("./huginn-eve-dsl");
|
||||
Object.defineProperty(exports, "buildHuginnEveDsl", { enumerable: true, get: function () { return huginn_eve_dsl_1.buildHuginnEveDsl; } });
|
||||
var inspection_1 = require("cultcache-ts/inspection");
|
||||
Object.defineProperty(exports, "inspectCultCacheBytes", { enumerable: true, get: function () { return inspection_1.inspectCultCacheBytes; } });
|
||||
70
package-lock.json
generated
70
package-lock.json
generated
|
|
@ -1,70 +0,0 @@
|
|||
{
|
||||
"name": "@gamecult/huginn",
|
||||
"version": "0.13.5",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@gamecult/huginn",
|
||||
"version": "0.13.5",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"cultcache-ts": "file:../CultLib/packages/cultcache-ts"
|
||||
},
|
||||
"bin": {
|
||||
"huginn": "dist/cli.js"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.0.0",
|
||||
"typescript": "^5.9.0"
|
||||
}
|
||||
},
|
||||
"../CultLib/packages/cultcache-ts": {
|
||||
"version": "0.13.5",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@msgpack/msgpack": "^3.1.3",
|
||||
"zod": "^3.25.76"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.0.0",
|
||||
"typescript": "^5.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "24.12.2",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.12.2.tgz",
|
||||
"integrity": "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~7.16.0"
|
||||
}
|
||||
},
|
||||
"node_modules/cultcache-ts": {
|
||||
"resolved": "../CultLib/packages/cultcache-ts",
|
||||
"link": true
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "5.9.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
||||
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.17"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "7.16.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.16.0.tgz",
|
||||
"integrity": "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
}
|
||||
}
|
||||
}
|
||||
35
package.json
35
package.json
|
|
@ -1,35 +0,0 @@
|
|||
{
|
||||
"name": "@gamecult/huginn",
|
||||
"version": "0.13.5",
|
||||
"description": "Huginn .cc inspection and Eve projection surface for CultCache state.",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"bin": {
|
||||
"huginn": "dist/cli.js"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"typecheck": "tsc -p tsconfig.json --pretty false",
|
||||
"test": "npm run build"
|
||||
},
|
||||
"keywords": [
|
||||
"messagepack",
|
||||
"schema",
|
||||
"cache",
|
||||
"persistence",
|
||||
"zod",
|
||||
"typescript"
|
||||
],
|
||||
"author": "metacratic",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"cultcache-ts": "file:../CultLib/packages/cultcache-ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.0.0",
|
||||
"typescript": "^5.9.0"
|
||||
}
|
||||
}
|
||||
11
schemas/cultnet/README.md
Normal file
11
schemas/cultnet/README.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# Huginn's CultNet contracts
|
||||
|
||||
`huginn.mind_request.v1` and `huginn.mind_response.v1` are the organ's own wire
|
||||
schemas, published from here because Huginn owns them. `huginn-daemon` serves
|
||||
them on a `cultnet.schema_catalog_request.v0`; nothing else reads them.
|
||||
|
||||
Each file is `serde_json::to_string_pretty` of `schemars::schema_for!` over the
|
||||
type of the same name in `crates/huginn-mind/src/wire.rs`. Regenerate by hand
|
||||
and the test refuses it:
|
||||
`cargo test -p huginn-mind --lib -- --exact wire::tests::published_wire_schemas_match_derivation`
|
||||
compares file to derivation byte for byte.
|
||||
1872
schemas/cultnet/huginn.mind_request.v1.schema.json
Normal file
1872
schemas/cultnet/huginn.mind_request.v1.schema.json
Normal file
File diff suppressed because it is too large
Load diff
3882
schemas/cultnet/huginn.mind_response.v1.schema.json
Normal file
3882
schemas/cultnet/huginn.mind_response.v1.schema.json
Normal file
File diff suppressed because it is too large
Load diff
23
src/cli.ts
23
src/cli.ts
|
|
@ -1,23 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { inspectCultCacheBytes } from "cultcache-ts/inspection";
|
||||
|
||||
import { buildHuginnEveDsl } from "./huginn-eve-dsl";
|
||||
|
||||
async function main(args: string[]): Promise<void> {
|
||||
const filePath = args[0];
|
||||
if (!filePath || args.includes("--help") || args.includes("-h")) {
|
||||
process.stderr.write("Usage: huginn <state.cc|state.msgpack|state.mpack>\n");
|
||||
process.exitCode = filePath ? 0 : 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const bytes = await readFile(filePath);
|
||||
const inspection = inspectCultCacheBytes(filePath, bytes);
|
||||
process.stdout.write(`${buildHuginnEveDsl(inspection)}\n`);
|
||||
}
|
||||
|
||||
void main(process.argv.slice(2)).catch((error: unknown) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
import type { CultCacheInspection, InspectedCatalogEntry, InspectedRecord } from "cultcache-ts/inspection";
|
||||
|
||||
export function buildHuginnEveDsl(inspection: CultCacheInspection): string {
|
||||
return [
|
||||
"# Huginn emits Eve DSL. Renderers compile and lower it.",
|
||||
`surface cultcache.huginn.inspector ${quoteDsl("Huginn .cc Inspector")}`,
|
||||
"version 1",
|
||||
"",
|
||||
"collection huginn.catalog.entries",
|
||||
...inspection.catalog.map((entry) => `item huginn.catalog.entries ${quoteDsl(renderCatalogEntry(entry))}`),
|
||||
"",
|
||||
"collection huginn.records.entries",
|
||||
...inspection.records.map((record) => `item huginn.records.entries ${quoteDsl(renderRecord(record))}`),
|
||||
"",
|
||||
"card source \"Source Witness\"",
|
||||
`text ${quoteDsl(`file: ${inspection.filePath}`)}`,
|
||||
`text ${quoteDsl(`format: ${inspection.format}`)}`,
|
||||
`text ${quoteDsl(`bytes: ${inspection.fileSizeBytes}`)}`,
|
||||
"text \".cc bytes are authority; Huginn only emits Eve DSL.\"",
|
||||
"end",
|
||||
"",
|
||||
"card catalog \"Schema Catalog\"",
|
||||
`text ${quoteDsl(`schemas: ${inspection.catalog.length}`)}`,
|
||||
"list \"Catalog Entries\" bind huginn.catalog.entries",
|
||||
"end",
|
||||
"",
|
||||
"card records \"Persisted Records\"",
|
||||
`text ${quoteDsl(`records: ${inspection.records.length}`)}`,
|
||||
"list \"Record Entries\" bind huginn.records.entries",
|
||||
"end",
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function renderCatalogEntry(entry: InspectedCatalogEntry): string {
|
||||
return [
|
||||
entry.schemaName,
|
||||
entry.schemaVersion,
|
||||
entry.schemaId,
|
||||
`${entry.members.length} members`,
|
||||
].join(" | ");
|
||||
}
|
||||
|
||||
function renderRecord(record: InspectedRecord): string {
|
||||
const status = record.payloadDecodeError ? `decode error: ${record.payloadDecodeError}` : "decoded";
|
||||
return [
|
||||
record.key,
|
||||
record.schemaName,
|
||||
record.storedAt,
|
||||
`${record.payloadBytes} bytes`,
|
||||
status,
|
||||
].join(" | ");
|
||||
}
|
||||
|
||||
function quoteDsl(value: string): string {
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
export { buildHuginnEveDsl } from "./huginn-eve-dsl";
|
||||
export { inspectCultCacheBytes } from "cultcache-ts/inspection";
|
||||
export type {
|
||||
CultCacheInspection,
|
||||
InspectedCatalogEntry,
|
||||
InspectedCatalogMember,
|
||||
InspectedRecord,
|
||||
} from "cultcache-ts/inspection";
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "Node16",
|
||||
"moduleResolution": "Node16",
|
||||
"lib": [
|
||||
"ES2022"
|
||||
],
|
||||
"strict": true,
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"rootDir": "src",
|
||||
"outDir": "dist"
|
||||
},
|
||||
"include": [
|
||||
"src/index.ts",
|
||||
"src/huginn-eve-dsl.ts",
|
||||
"src/cli.ts"
|
||||
],
|
||||
"exclude": [
|
||||
"test/**/*.ts"
|
||||
]
|
||||
}
|
||||
Loading…
Reference in a new issue