From 91cff588582fe6d85bd1f52d2eaf0a8254374f4c Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 19 May 2026 13:00:08 +0100 Subject: [PATCH] Type Qdrant retrieval payloads --- epiphany-core/src/retrieval.rs | 38 ++++++++++++++++++------------- notes/fresh-workspace-handoff.md | 1 + state/ledgers.msgpack | Bin 131312 -> 131643 bytes state/map.yaml | 2 +- 4 files changed, 24 insertions(+), 17 deletions(-) diff --git a/epiphany-core/src/retrieval.rs b/epiphany-core/src/retrieval.rs index 1737fcfe..38f0f3af 100644 --- a/epiphany-core/src/retrieval.rs +++ b/epiphany-core/src/retrieval.rs @@ -392,12 +392,12 @@ fn index_workspace_with_config( .map(|(chunk, vector)| QdrantPointInput { id: chunk_point_id(&expected_collection, &chunk), vector, - payload: json!({ - "path": normalize_payload_path(&chunk.path), - "line_start": chunk.line_start, - "line_end": chunk.line_end, - "excerpt": chunk.excerpt, - }), + payload: QdrantPointPayload { + path: normalize_payload_path(&chunk.path), + line_start: chunk.line_start, + line_end: chunk.line_end, + excerpt: chunk.excerpt, + }, }) .collect::>(); qdrant.upsert_points(&expected_collection, &points)?; @@ -1022,7 +1022,16 @@ impl OllamaEmbedder { struct QdrantPointInput { id: String, vector: Vec, - payload: Value, + payload: QdrantPointPayload, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +struct QdrantPointPayload { + path: String, + line_start: u32, + line_end: u32, + excerpt: String, } #[derive(Debug, Deserialize)] @@ -1044,7 +1053,7 @@ struct QdrantQueryResultEnvelope { struct QdrantQueryPoint { score: f32, #[serde(default)] - payload: BTreeMap, + payload: Option, } #[derive(Debug, Deserialize)] @@ -1386,18 +1395,15 @@ fn search_semantic_chunks( } fn map_qdrant_hit_to_result(point: QdrantQueryPoint) -> Option { - let path = point.payload.get("path")?.as_str()?; - let excerpt = point.payload.get("excerpt")?.as_str()?.to_string(); - let line_start = point.payload.get("line_start")?.as_u64()? as u32; - let line_end = point.payload.get("line_end")?.as_u64()? as u32; + let payload = point.payload?; Some(EpiphanyRetrieveResult { kind: EpiphanyRetrieveResultKind::SemanticChunk, - path: PathBuf::from(path), + path: PathBuf::from(payload.path), score: point.score, - line_start: Some(line_start), - line_end: Some(line_end), - excerpt: Some(excerpt), + line_start: Some(payload.line_start), + line_end: Some(payload.line_end), + excerpt: Some(payload.excerpt), }) } diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 5357ec2a..187de789 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -42,6 +42,7 @@ remembers doctrine; the branch remembers the blade. - The native state ledger is closed typed state now: branch and evidence records no longer preserve flattened `extra` maps. Evidence entries are explicit `ts` / `type` / `status` / `note` / optional branch records. - `epiphany-openai-codex-spine` no longer uses Codex's Responses client, model-provider layer, `codex-api`, `codex-model-provider`, `codex-model-provider-info`, or `codex-protocol` for outbound model requests. It reaches `codex-login` only through `epiphany-openai-auth-spine` so credential loading/refresh stays Codex-compatible while request/status/event/runtime surfaces stay typed Epiphany documents. - `notes/epiphany-memory-graph-unified-plan.md` is the current memory architecture direction drawn from `Fractal Domains And The Cache That Bites` plus the VoidBot memory scar tissue. The repo semantic architecture/dataflow graph and agent memory are one `EpiphanyMemoryGraph` substrate with repo, role, short-term, incubation, agency, candidate-intervention, and evidence profiles. The first native skeleton is landed: `epiphany-state-model` owns shared graph document types, `epiphany-core::memory_graph` owns stable ids, validation, freshness, context cuts, and CultCache persistence, and `epiphany-memory-graph` can status/validate/context/smoke the store without Qdrant. Qdrant is a contribution cache for typed graph documents, not truth. `notes/repo-fractal-dataflow-cache-plan.md` and `notes/agent-memory-fractal-cache-plan.md` are profile plans, not rival organs. +- Retrieval's Qdrant boundary has a typed point payload now. Qdrant's HTTP requests, responses, and acknowledgements are JSON wire edges; Epiphany's retrieval facts (`path`, `line_start`, `line_end`, `excerpt`) are `QdrantPointPayload` before serialization/deserialization, not a loose payload map. - Durable Epiphany state types now live in root crate `epiphany-state-model`. `codex-protocol` re-exports them only so old rollout/API callers keep compiling; do not add new durable `Epiphany*` state structs under `vendor/codex`. The deterministic `` prompt renderer also lives beside that typed model now, with prompt text under `epiphany-state-model/src/prompts/`; `epiphany-core` only re-exports it for compatibility. `epiphany-core` imports the native model directly and no longer depends on `codex-protocol`. Pressure derives from native `EpiphanyTokenUsageSnapshot`; Codex token telemetry is mapped at the bridge. Codex rollout/event reconstruction now lives in `codex-core::epiphany_rollout`, where the Codex `RolloutItem` / `EventMsg` types belong. Exact retrieval path matching is native to `epiphany-core`, so `epiphany-core` has no direct vendored Codex crate dependency. - State-update contract proof and job launch/interrupt DTO ownership now live in native `epiphany-core`. The old giant `codex_thread.rs` Epiphany state-update test fixture was cut, and `epiphany-codex-bridge` no longer imports Epiphany job launch request types from `codex_core`; vendored `CodexThread` may call the native mutation/launch law for compatibility, but it must not regain ownership of state-update semantics, heartbeat launch-plan policy, or Epiphany DTOs. - MCP remains JSON because MCP is a JSON protocol. The latest processor cut only moved MCP refresh/OAuth/status/resource/tool-call route handling into `codex_message_processor/mcp_routes.rs`, reducing the root processor to about 8,977 lines; it did not claim MCP payloads are CultNet or make MCP an Epiphany-owned runtime contract. diff --git a/state/ledgers.msgpack b/state/ledgers.msgpack index ce8dfe6cc27b5f9a0a8ccbecfcecea08f6014d17..53534a47623a5e1a949cb2394f53f8d2a8749502 100644 GIT binary patch delta 357 zcmZ9I!AiqG6h%o_UAqzryll4WDA-Oq?Z@hE}ww z$r7!iq*6nv3~9SND@oT-pi*39z=g5ovdR6g~oK>{%;oU2e&`{k9}zX delta 48 zcmdnp!ttSzV}ddh!?um8g^Y}4&C?mTPiJIw`7gS~$iT=<*T7WQ&@#l((8|QZ%G5Fn E0Fw$1vj6}9 diff --git a/state/map.yaml b/state/map.yaml index fa425b3c..2196f21a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -115,7 +115,7 @@ accepted_design: - Durable typed state writes route through `thread/epiphany/update`, accepted `thread/epiphany/promote`, `thread/epiphany/reorientAccept`, `thread/epiphany/roleAccept`, or the bounded `thread/epiphany/jobLaunch`, `thread/epiphany/jobInterrupt`, `thread/epiphany/reorientLaunch`, and `thread/epiphany/roleLaunch` authority surfaces when they mutate state or `job_bindings`. - `thread/epiphany/retrieve`, `thread/epiphany/distill`, `thread/epiphany/propose`, `thread/epiphany/freshness`, `thread/epiphany/context`, `thread/epiphany/graphQuery`, and `thread/epiphany/view` stay read-only. - `thread/epiphany/index` writes the retrieval catalog, not durable Epiphany understanding; catalog is auspex output, not doctrine. - - Qdrant is the preferred persistent semantic backend; BM25 remains the bootstrap/fallback/control path. + - Qdrant is the preferred persistent semantic backend; BM25 remains the bootstrap/fallback/control path. Qdrant's HTTP API and acknowledgements remain JSON wire edges, but Epiphany-owned retrieval point payloads are typed at the adapter boundary before serialization/deserialization. - `notes/epiphany-memory-graph-unified-plan.md` applies the mapped-fractal-domain/cache lesson to Epiphany and corrects the organ boundary: repo architecture/dataflow and agent memory are profiles over one `EpiphanyMemoryGraph`, not separate graph engines. The first native skeleton is landed: `epiphany-state-model` owns the public document vocabulary, `epiphany-core::memory_graph` owns stable ids/validation/freshness/context cuts/CultCache persistence, and `epiphany-memory-graph` exposes status/validate/context/smoke without Qdrant. `notes/repo-fractal-dataflow-cache-plan.md` and `notes/agent-memory-fractal-cache-plan.md` are profile plans beneath that substrate. If deleting Qdrant deletes understanding, the design is wrong; the fallback must be slower but still correct typed graph traversal. - `epiphany-state-model` owns the deterministic prompt-state renderer beside the typed `EpiphanyThreadState` document it serializes. `epiphany-core` re-exports that renderer for compatibility and owns rollout replay helpers, retrieval/indexing, distillation, proposal, promotion policy, scene projection, freshness derivation, pressure derivation, job/progress view derivation, planning view derivation, reorientation resume/regather verdict policy, pure CRRC recommendation policy, fixed-lane coordinator decision policy, role-board projection policy, and role-result interpretation/self-persistence review policy where possible. - Vendored Codex keeps the thin typed host seam: protocol types, thread bridges, app-server protocol, request routing, watcher lifecycle, notifications, rollout integration, and host persistence facts such as current Epiphany state, reference turn id, workspace root, Codex home, and runtime store path. Vendored `codex-core` and `codex-app-server` must not import `epiphany-core` directly; Epiphany update law, retrieval/indexing, launch request construction, and heartbeat launch/interrupt policy belong in `epiphany-codex-bridge` / `epiphany-core`.